ProSoundWeb Community

Sound Reinforcement - Forums for Live Sound Professionals - Your Displayed Name Must Be Your Real Full Name To Post In The Live Sound Forums => LAB: The Classic Live Audio Board => Topic started by: Tim Weaver on March 08, 2021, 09:24:35 PM

Title: New and exciting Dante frustrations.
Post by: Tim Weaver on March 08, 2021, 09:24:35 PM
I added two new devices to my network this week and one of them broke things.

Device one is a new iMac running 10.14.6 Mojave and DVS (newest build). My 2012 iMac finally bought the farm so I had to replace it with this machine we had on hand. It is used to record using Reaper, plus various other management tasks.

Device two is a custom built PC running Win10. This has a Rednet PCIeR card in it to route audio in and out of LiveProfessor so I can host plug ins on my desk. The desk (CL5) uses an MY16 Dante card for the inserts.


The iMac works fine until I route audio to the DSP Host PC. This is an important distinction to note, The PC can sit on the network and cause no issues whatsoever. But when I route audio to the PC, the iMac starts losing clock sync.


I have tried the onboard ethernet, and a tbolt ethernet dongle on the mac. Same thing happens.



Also when I route audio to the DSP host I get jumping clock masters in Dante. I have no preffered master set so Dante can choose whatever it wants. When I start using the DSP the master starts jumping from one device to the other. Sometimes I will lose audio through the PC which cuts audio on my desk since the plugs are inserted, but nothing else is affected it seems.

I have wired the PC as many different ways as I can figure out, but it still keeps happening. The crappy thing is that Dante Controller gives me no errors about what is happening with the Clock master jumping around. The only error I get is the iMac losing clock sync over and over.


Any ideas?
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 08, 2021, 09:29:41 PM
BTW an interesting thing I learned while experimenting is that if you have an MY16 Dante card in your desk it will show up on Dante Controller whether it is connected by ethernet or not. You can't actually airgap it from the main dante network. This is kinda frutrating because I was hoping to do a link-local connection between the MY card and the Rednet PCIeR. It wouldn't work because my MY card was getting DHCP from the Dante network through the desk.

Also it's interesting that if you just run a cat cable from the MY card to the PC you can't actually see the PC on Dante controller, but you can see the MY card. This is the same whether it is in redundant or switched mode. So the desk is passing network through to the MY card, but that card won't let you see anything past it's ports.
Title: Re: New and exciting Dante frustrations.
Post by: Russell Ault on March 08, 2021, 09:59:59 PM
I added two new devices to my network this week and one of them broke things.

Device one is a new iMac running 10.14.6 Mojave and DVS (newest build). My 2012 iMac finally bought the farm so I had to replace it with this machine we had on hand. It is used to record using Reaper, plus various other management tasks.

Device two is a custom built PC running Win10. This has a Rednet PCIeR card in it to route audio in and out of LiveProfessor so I can host plug ins on my desk. The desk (CL5) uses an MY16 Dante card for the inserts.


The iMac works fine until I route audio to the DSP Host PC. This is an important distinction to note, The PC can sit on the network and cause no issues whatsoever. But when I route audio to the PC, the iMac starts losing clock sync.


I have tried the onboard ethernet, and a tbolt ethernet dongle on the mac. Same thing happens.



Also when I route audio to the DSP host I get jumping clock masters in Dante. I have no preffered master set so Dante can choose whatever it wants. When I start using the DSP the master starts jumping from one device to the other. Sometimes I will lose audio through the PC which cuts audio on my desk since the plugs are inserted, but nothing else is affected it seems.

I have wired the PC as many different ways as I can figure out, but it still keeps happening. The crappy thing is that Dante Controller gives me no errors about what is happening with the Clock master jumping around. The only error I get is the iMac losing clock sync over and over.


Any ideas?

Just so I'm clear, despite what you'd had hoped to to, the Mac, the Windows machine, and all of the Dante ports on the CL5 are now on the same network, correct?

DVS clock sync issues in OS 10.14 appears to be a known issue, although a Thunderbolt port is suggested as a workaround which I know you've tried.

When you say "jumping around", do you mean that the Master Clock is cycling back and forth between devices all on its own, or do you mean that the Master Clock changes to the PCIe card when you start sending flows to it?

Have you tried replacing the network cable between the PCIe card and the switch? Can you downgrade the Mac to 10.12? Is there a reason you don't want to set a Preferred Master clock?

-Russ
Title: Re: New and exciting Dante frustrations.
Post by: David Sturzenbecher on March 08, 2021, 10:05:09 PM
BTW an interesting thing I learned while experimenting is that if you have an MY16 Dante card in your desk it will show up on Dante Controller whether it is connected by ethernet or not. You can't actually airgap it from the main dante network. This is kinda frutrating because I was hoping to do a link-local connection between the MY card and the Rednet PCIeR. It wouldn't work because my MY card was getting DHCP from the Dante network through the desk.

Also it's interesting that if you just run a cat cable from the MY card to the PC you can't actually see the PC on Dante controller, but you can see the MY card. This is the same whether it is in redundant or switched mode. So the desk is passing network through to the MY card, but that card won't let you see anything past it's ports.

That is pretty wild for sure.  I have only installed a MY Dante in CL once, but never went through a similar process.  Could you set the IP of the MY card and the Rednet PCIeR to static in a range outside the DHCP assigned subnet?   You would likely get a Red warning in dante controller, and I have no idea how the clocks would work then...

I don't use a Macs regularly, but I know there are some issues reported with one of the OSs over on the Dante Facebook group.  Unfortunately, i haven't been paying enough attention to know any more details, but it may be worth a search over there.   
Title: Re: New and exciting Dante frustrations.
Post by: Nathan Salt on March 08, 2021, 10:26:52 PM
Sorry I have nothing to add to help fix the problem. I can only add that we have different issues with our Rednet PCIeR card.
We have one in a Vmix machine, and connected normally to our dante network. When mounted to our CL3 it wipes all dante patching on boot up of the computer....
ALL input patching, not just from the card.
Our work around is to not have it mounted in the console and patch it from controller. Seems to be working so far

I wouldn't be surprised if it was the Rednet card causing you issues too.

We also have a Dante MY card hooked up to insert noise reduction over mics. Running off a little nuc with Carla hosting the VSTs. All works flawlessly atm

Keen to see where this thread goes
Title: Re: New and exciting Dante frustrations.
Post by: Russell Ault on March 08, 2021, 10:31:58 PM
{...} Any ideas?

Something else just twigged in my mind: is the Dante firmware up-to-date on all the device (bearing in mind that the Dante firmware is typically separate from the device's firmware)?

-Russ
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 08, 2021, 10:54:18 PM
Something else just twigged in my mind: is the Dante firmware up-to-date on all the device (bearing in mind that the Dante firmware is typically separate from the device's firmware)?

-Russ

Probably not. I will do that, but I couldn't Sunday or Monday. Now I have until this Thursday to either fix it or nuke the DSP so I can do another weekend of services.

This system has been stable for a while and so I haven't touched anything in probably two years.
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 08, 2021, 11:02:36 PM
Just so I'm clear, despite what you'd had hoped to to, the Mac, the Windows machine, and all of the Dante ports on the CL5 are now on the same network, correct?
Yes, all on a Vlan that exists just for Dante
Quote

DVS clock sync issues in OS 10.14 appears to be a known issue, although a Thunderbolt port is suggested as a workaround which I know you've tried.
Yes. Didn't change anything
Quote


When you say "jumping around", do you mean that the Master Clock is cycling back and forth between devices all on its own, or do you mean that the Master Clock changes to the PCIe card when you start sending flows to it?
Yes, it's jumping among all the dante devices on the network. Typically one of my Shure wireless units is the master. I assume it has the lowest MAC address, since it is always the master. Today Master Clock was switching off every 30 seconds to 2 minutes to new devices.
Quote


Have you tried replacing the network cable between the PCIe card and the switch? Can you downgrade the Mac to 10.12? Is there a reason you don't want to set a Preferred Master clock?

-Russ
I tried different cables.

I also tried to set the console as the master. It didn't matter and would jump anyway. Often there would be two "Masters" shown in Dante Controller at the same time.

On a suggestion from another thread I tried to set the desk to internal clock, then the MY card as the master and set the MY card as "external clock source". That didn't work either. Thinking it through now, I can see why that didn't work. The desk also is on the network, and would still need to be clocked from Dante.
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 08, 2021, 11:05:20 PM
That is pretty wild for sure.  I have only installed a MY Dante in CL once, but never went through a similar process.  Could you set the IP of the MY card and the Rednet PCIeR to static in a range outside the DHCP assigned subnet?   You would likely get a Red warning in dante controller, and I have no idea how the clocks would work then...
I can try that, but I'll need to recruit my IT guy to carve out that space for me. I can't just do it myself.
Quote
I don't use a Macs regularly, but I know there are some issues reported with one of the OSs over on the Dante Facebook group.  Unfortunately, i haven't been paying enough attention to know any more details, but it may be worth a search over there.   
I will give it a go.
Title: Re: New and exciting Dante frustrations.
Post by: David Sturzenbecher on March 08, 2021, 11:13:06 PM
Yes, all on a Vlan that exists just for Dante  Yes. Didn't change anything Yes, it's jumping among all the dante devices on the network. Typically one of my Shure wireless units is the master. I assume it has the lowest MAC address, since it is always the master. Today Master Clock was switching off every 30 seconds to 2 minutes to new devices.I tried different cables.

I also tried to set the console as the master. It didn't matter and would jump anyway. Often there would be two "Masters" shown in Dante Controller at the same time.

On a suggestion from another thread I tried to set the desk to internal clock, then the MY card as the master and set the MY card as "external clock source". That didn't work either. Thinking it through now, I can see why that didn't work. The desk also is on the network, and would still need to be clocked from Dante.

In Dante controller, for your event log, on the lower left, you can set it to “information” it might give you some more details. 

I have a site with a portable rack that is doing something similar. When plugged in, that rack will be promoted master, then demoted, once every 30-120 seconds. It has zero affect on audio though.  My issue only happens when connected over fiber and not cat5.  I suspect I may have a bad SFP, but I need to do some more investigating.
Title: Re: New and exciting Dante frustrations.
Post by: Russell Ault on March 09, 2021, 12:06:46 AM
Probably not. I will do that, but I couldn't Sunday or Monday. Now I have until this Thursday to either fix it or nuke the DSP so I can do another weekend of services.

This system has been stable for a while and so I haven't touched anything in probably two years.

I'd start here then, for sure; don't forget to update the Dante firmware on the PCIe card as well.

Good luck!

-Russ
Title: Re: New and exciting Dante frustrations.
Post by: lindsay Dean on March 09, 2021, 01:17:17 PM
 set the CL as your master clock
All other Dante devices need to be set to slave
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 09, 2021, 02:16:22 PM
set the CL as your master clock
All other Dante devices need to be set to slave

That didn't work. It was still jumping around. As far as I understand, setting a device as Preferred master is just that. A preference. Dante can still vote that device off the island and do what it needs to do.
Title: Re: New and exciting Dante frustrations.
Post by: Jon Dees on March 09, 2021, 04:51:48 PM
That didn't work. It was still jumping around. As far as I understand, setting a device as Preferred master is just that. A preference. Dante can still vote that device off the island and do what it needs to do.

Was also going to suggest setting your desk as preferred master. I do have a PCIeR, multiple DVS instances (1 Mojave), QL5, Tios, x32, Blu-DA, and more on my Dante network. It has been stable in this configuration since November, at least. The last device we added was the PCIeR and no issues with integration. We are running Cisco and Netgear switches. You might try different network switches or complicated network diagnostics.

Everything did need to be in the Dante version 4.0.9 or higher to be happy. If anything is on Dante version 3 I would start with that first.
Title: Re: New and exciting Dante frustrations.
Post by: Ron Bolte on March 09, 2021, 06:52:11 PM
BTW an interesting thing I learned while experimenting is that if you have an MY16 Dante card in your desk it will show up on Dante Controller whether it is connected by ethernet or not. You can't actually airgap it from the main dante network. This is kinda frutrating because I was hoping to do a link-local connection between the MY card and the Rednet PCIeR. It wouldn't work because my MY card was getting DHCP from the Dante network through the desk.

Also it's interesting that if you just run a cat cable from the MY card to the PC you can't actually see the PC on Dante controller, but you can see the MY card. This is the same whether it is in redundant or switched mode. So the desk is passing network through to the MY card, but that card won't let you see anything past it's ports.

Hi Tim, Im running a MY card in both a QL5 and CL5 and can't get them to display this behavior.  When I pull the connection from either, they drop off of the Dante network.  Maybe this weird behavior is near the root of your issues?
Title: Re: New and exciting Dante frustrations.
Post by: Phillip Ivan Pietruschka on March 09, 2021, 08:53:20 PM
Hi Tim, Im running a MY card in both a QL5 and CL5 and can't get them to display this behavior.  When I pull the connection from either, they drop off of the Dante network.  Maybe this weird behavior is near the root of your issues?

I have never observed this behaviour either, and indicates to me there is something else going on. I have seen some network switches drop multicast packets which can interfere with all aspects of Dante (PTP, device discovery, transport); with a small network the easiest way to test it for this is to take the devices onto a separate switch.

I’d strongly recommend keeping firmware up to date and setting a preferred master as a matter of course.
Title: Re: New and exciting Dante frustrations.
Post by: David Sturzenbecher on March 09, 2021, 09:39:11 PM
I have never observed this behaviour either, and indicates to me there is something else going on. I have seen some network switches drop multicast packets which can interfere with all aspects of Dante (PTP, device discovery, transport); with a small network the easiest way to test it for this is to take the devices onto a separate switch.

I’d strongly recommend keeping firmware up to date and setting a preferred master as a matter of course.

Tim,
When you disconnected the cables from the MY card, can you confirm you did to both the primary and secondary. We are all assuming so, but it wasn't explicitly stated.
Title: Re: New and exciting Dante frustrations.
Post by: Tim McCulloch on March 09, 2021, 09:53:32 PM
What do the logs say?  There should be entries for every time a device is recognized and what it's clock status is after playing the "elect my 'clock leader'."
Title: Re: New and exciting Dante frustrations.
Post by: Glen Kelley on March 10, 2021, 03:15:58 PM
What do the logs say?  There should be entries for every time a device is recognized and what it's clock status is after playing the "elect my 'clock leader'."

Tim W, I saw this behavior in a system with Dante Via running on PC, a Q-sys core and few other devices. The clock would bounce between the core and the computer running Dante Via. In the logs, you would see an endless loop of devices being elevated to Grandmaster.

In our case, the issue ended up being the Q-sys PTP Priority 1 set too 'low' (ie. a higher number). The fix was to change the PTP priority in the Core design from 254 to something like 150. (I don't remember the actual value, but it was a lower number than the default)

If you can set this on the new devices that broke things, see if one of them might have the PTP Priority 1 set to a level that makes it look like a good Master Clock when it should not be. A higher number moves that device 'down' the pecking order for clock selection.
Title: Re: New and exciting Dante frustrations.
Post by: Nathan Schwarzkopf on March 18, 2021, 08:51:19 AM
Tim,

It sounds to me like something is arriving late.  I had all kinds of problems with this lately.  It wont throw errors in DC but it will drop audio so you dont hear the clicks and pops.  There is a latency tab in DC to see when the packets arrive, I would check there first. Are you going point to point or are you going through a switch?  If you are going through a switch and it is managed, it could be a set up thing, or a set up thing on specific ports.  I also noticed mention of a core, is the QOS set to audinate?  Good luck, this can be a PITA.

Nathan
Title: Re: New and exciting Dante frustrations.
Post by: Steve-White on March 26, 2021, 07:57:56 PM
Tim,

It sounds to me like something is arriving late.  I had all kinds of problems with this lately.  It wont throw errors in DC but it will drop audio so you dont hear the clicks and pops.  There is a latency tab in DC to see when the packets arrive, I would check there first. Are you going point to point or are you going through a switch?  If you are going through a switch and it is managed, it could be a set up thing, or a set up thing on specific ports.  I also noticed mention of a core, is the QOS set to audinate?  Good luck, this can be a PITA.

Nathan

Right you are sir.  And that could be from a myriad of causes ranging from hardware/firmware integration or even a hardware fault like a weak output driver to the wiring system in the building.

Troubleshooting could be a real adventure.  If possible, I'd be tempted to some temporary bypassing of all the interconnecting cables for some testing just to be sure.  Or get someone out there that can do proper impedance sweeps and TDR ring out of the cabling system.

In addition it was mentioned to search Facebook for known problems - that may end things before they start.
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 26, 2021, 09:36:33 PM
Right you are sir.  And that could be from a myriad of causes ranging from hardware/firmware integration or even a hardware fault like a weak output driver to the wiring system in the building.

Troubleshooting could be a real adventure.  If possible, I'd be tempted to some temporary bypassing of all the interconnecting cables for some testing just to be sure.  Or get someone out there that can do proper impedance sweeps and TDR ring out of the cabling system.

In addition it was mentioned to search Facebook for known problems - that may end things before they start.

Just as an update, I haven't solved all of this yet. I've been working through the updates in my limited time windows which I can take the rig offline, but I'm having a real fight with updating my rio boxes. I will call Yamaha and sit on the phone with them next week and hopefully get the firmware all sorted. My whole rig was on Dante 3.0.8 or something like that and had been for a while.

The MY card turns out was not recieving dhcp through the console. That was a mispatch on my part. It's in a difficult spot to reach and I was unplugging it at the switch instead of crawling under my desk. Turns out I was not unplugging the MY card.

I found a mispatch where one secondary port was patched back into a primary port. This normally isn't a problem since I'm not running redundant networks, but this particular secondary port can not be put into switch mode, so it was no bueno. This mispatch didn't seem to be causing any troubles though. Still I'm glad I found it.

I still have the issue of a new clock master being elevated every 30-120 seconds.


Another problem I have is that I have no control over the network. I have an IT guy who manages that and he's tough to work with. Next week I'll be looking at the QOS settings with him as I've read another person had an issue with a device with a too-high QOS setting trying to always be the master.
Title: Re: New and exciting Dante frustrations.
Post by: Steve-White on March 26, 2021, 11:33:09 PM
^^^ Thanks for the update.  Just for the record, I have "0" experience with Dante.

But have background with some computer architecture and integrated systems and networks are networks.  :)

Just trying to help out.  My background to date is defense systems avionics and automatic test equipment to be honest with ya.

But now moving into state of the art audio system applications.  I talked to a consultant earlier in the week and Dante came up centered around DSC's, FIR filters and such.  I was into pro audio before things "went digital" and after a career in the aircraft industry am moving back into audio.

Having the background and experience gained, it's still a formidable leap forward.  A guy darn near needs network architecture and engineering experience to even touch today's equipment.

Please do post the journey and results.
Title: Re: New and exciting Dante frustrations.
Post by: Phillip Ivan Pietruschka on March 27, 2021, 05:18:59 AM
Just as an update, I haven't solved all of this yet. I've been working through the updates in my limited time windows which I can take the rig offline, but I'm having a real fight with updating my rio boxes. I will call Yamaha and sit on the phone with them next week and hopefully get the firmware all sorted. My whole rig was on Dante 3.0.8 or something like that and had been for a while.

The MY card turns out was not recieving dhcp through the console. That was a mispatch on my part. It's in a difficult spot to reach and I was unplugging it at the switch instead of crawling under my desk. Turns out I was not unplugging the MY card.

I found a mispatch where one secondary port was patched back into a primary port. This normally isn't a problem since I'm not running redundant networks, but this particular secondary port can not be put into switch mode, so it was no bueno. This mispatch didn't seem to be causing any troubles though. Still I'm glad I found it.

I still have the issue of a new clock master being elevated every 30-120 seconds.


Another problem I have is that I have no control over the network. I have an IT guy who manages that and he's tough to work with. Next week I'll be looking at the QOS settings with him as I've read another person had an issue with a device with a too-high QOS setting trying to always be the master.

Tim with the firmware and patching issues solved, I would suggest, if practical, moving the Dante devices to a separate basic switch (non otherwise connected to the larger corporate network), and see if the problems persist. The answer to this should give you a very clear indication of if the configuration or behaviour of the corporate switch is the problem or if the fault lies elsewhere (configuration, faulty device, faulty cables).

For the relatively small number of devices that seem to be involved, it should not be necessary for the test switch to have QOS or IGMP configured.
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 27, 2021, 09:56:12 PM
Tim with the firmware and patching issues solved, I would suggest, if practical, moving the Dante devices to a separate basic switch (non otherwise connected to the larger corporate network), and see if the problems persist. The answer to this should give you a very clear indication of if the configuration or behaviour of the corporate switch is the problem or if the fault lies elsewhere (configuration, faulty device, faulty cables).

For the relatively small number of devices that seem to be involved, it should not be necessary for the test switch to have QOS or IGMP configured.

If a fully up-to-date set of firmwares doesn't fix it, I will be moving on to that as a method of troubleshooting.

My biggest issue is that Easter is coming (this is a church system) and I have limited windows in which I can really take the system offline. Wednesday and Thursday basically, but I work (youth service) in a different venue on wednesday night. So most of my big stuff happens on Thursdays.
Title: Re: New and exciting Dante frustrations.
Post by: Phillip Ivan Pietruschka on March 28, 2021, 02:14:38 AM
Tim, if you do need to resume the conversation with your IT team, the following document will be essential:
https://www.audinate.com/learning/faqs/which-network-ports-does-dante-use

Also, I recommend downloading PTP trackhound which is like Wireshark optimised for PTP monitoring.
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on March 28, 2021, 09:57:04 AM
Tim, if you do need to resume the conversation with your IT team, the following document will be essential:
https://www.audinate.com/learning/faqs/which-network-ports-does-dante-use

Also, I recommend downloading PTP trackhound which is like Wireshark optimised for PTP monitoring.

Thanks for this. I am not an IT professional™ so things like this helps tremendously.
Title: Re: New and exciting Dante frustrations.
Post by: Tim Weaver on June 07, 2021, 08:36:03 PM
I thought I'd come back with an update on this.

I'm not sure why or how but someday recently I fired up Dante Controller and noticed a pending update. It was for my MY16 card. Updated it and no morre problems! lol

I guess it was just the right recipe of firmware mismatches to cause the clock sync to randomly jump around. I don't know, but it's working again. I finally have plug ins on my desk again which is awesome.
Title: Re: New and exciting Dante frustrations.
Post by: Jim McKeveny on June 10, 2021, 08:34:38 AM
I thought I'd come back with an update on this.

I'm not sure why or how but someday recently I fired up Dante Controller and noticed a pending update. It was for my MY16 card. Updated it and no morre problems! lol

I guess it was just the right recipe of firmware mismatches to cause the clock sync to randomly jump around. I don't know, but it's working again. I finally have plug ins on my desk again which is awesome.

Another day in digital-perfect world! ("digital perfect" was an early selling point for the Compact Disc, which proved neither perfect nor permanent).