ProSoundWeb Community
Sound Reinforcement - Forums for Live Sound Professionals - Your Displayed Name Must Be Your Real Full Name To Post In The Live Sound Forums => Pro AV Forum => Topic started by: Derek Toews on March 30, 2018, 11:55:56 AM
-
I've tried troubleshooting this with Audinate multiple times but we haven't been able to figure out what's going on so any help would be greatly appreciated!
I’ve got a Cisco SG300 (Audinate’s recommended switch) and I went through all the settings per the Yamaha manual so it’s setup properly (Been over all kinds of other settings beyond Yamaha’s manual with Audinate).
Running 2 Yamaha CL5’s, 2 RIO’s, 3 ULXD units, X32 rack, and some computers with DVS. Everything is updated to latest Dante firmware, latest DVS software, etc.
- Basically the problem is that the computers on the network will get glitches of high latency (above 20ms) but it’s only for a split second, and it only happens every once in a while (very random). And depending on how much data is being received by the computer depends on how much of a latency glitch there will be. (Only 5 multicast flows to a computer - it gets about 6-8 msec of latency; 20 multicast flows - 20+ msec of latency).
When multi tracking and it gets a latency “glitch" - it causes the computer to lose sync (even though Dante doesn’t show any sync errors or packet errors). When it loses sync, the multitrack also glitches pretty bad till it sync's up again (only a few seconds typically). Occasionally a computer will even act as if it was kicked off the Dante network (it will still receive flows from all the other computers on the network - but it stops receiving flows from Consoles / RIO’s / ULXDs). DVS shows it's still working though and there's no errors anywhere.
I've swapped all ethernet cables to computers, RIOs, CL5's. Purchased another Cisco SG300 switch. Everything is on a redundant network and both switches are the same Cisco SG300-28.
Any help would be greatly appreciated!
-
What else is on the neywork? I’m talking about non-dante stuff. Wireless AP’s, computers, other switches, fiber hops, etc.
We just went throught something similar and it was our new WAP’s that were installed. They couldn’t deal with the multicast traffic and basically shut down the network.
-
Nothing else is on the network - its all a completely closed Dante network. I’ve tried hooking it into our main network on a separate VLan and Meraki switches but had the same issue.
What else is on the neywork? I’m talking about non-dante stuff. Wireless AP’s, computers, other switches, fiber hops, etc.
We just went throught something similar and it was our new WAP’s that were installed. They couldn’t deal with the multicast traffic and basically shut down the network.
-
In dante controller is there a device or two that are checked as “preferred clock master”?
I know Dante is good about doing this itself, but I still want a console to be the clock master. If nothing else, I know the console has a pretty robist power supply and shouldn’t be effected by power anomalies.
-
I've tried troubleshooting this with Audinate multiple times but we haven't been able to figure out what's going on so any help would be greatly appreciated!
I’ve got a Cisco SG300 (Audinate’s recommended switch) and I went through all the settings per the Yamaha manual so it’s setup properly (Been over all kinds of other settings beyond Yamaha’s manual with Audinate).
Running 2 Yamaha CL5’s, 2 RIO’s, 3 ULXD units, X32 rack, and some computers with DVS. Everything is updated to latest Dante firmware, latest DVS software, etc.
- Basically the problem is that the computers on the network will get glitches of high latency (above 20ms) but it’s only for a split second, and it only happens every once in a while (very random). And depending on how much data is being received by the computer depends on how much of a latency glitch there will be. (Only 5 multicast flows to a computer - it gets about 6-8 msec of latency; 20 multicast flows - 20+ msec of latency).
When multi tracking and it gets a latency “glitch" - it causes the computer to lose sync (even though Dante doesn’t show any sync errors or packet errors). When it loses sync, the multitrack also glitches pretty bad till it sync's up again (only a few seconds typically). Occasionally a computer will even act as if it was kicked off the Dante network (it will still receive flows from all the other computers on the network - but it stops receiving flows from Consoles / RIO’s / ULXDs). DVS shows it's still working though and there's no errors anywhere.
I've swapped all ethernet cables to computers, RIOs, CL5's. Purchased another Cisco SG300 switch. Everything is on a redundant network and both switches are the same Cisco SG300-28.
Any help would be greatly appreciated!
I certainly can not be sure if this is your problem but I wanted to make people aware.
There were SG300's that required a Cisco update or mod of some type due to a problem that caused glitching with QLan from QSYS (the reason the SG300 was not approved by QSC). This glitch did not go back to the beggining of Yamaha's recommendating the SG300 series but started to occur in some units just a few years ago. Cisco was unable or unwilling to put out any information on a version/revision/lot number, etc., to help identify potential problem units but they did accept the problem units back and performed some type of update or service that corrected the problem. This was done by Cisco on a unit by unit basis.
I do not know if this glitch also affected Dante traffic. That said, I have not had an SG300 that has exhibited this problem.
Lee
-
I certainly can not be sure if this is your problem but I wanted to make people aware.
There were SG300's that required a Cisco update or mod of some type due to a problem that caused glitching with QLan from QSYS (the reason the SG300 was not approved by QSC). This glitch did not go back to the beggining of Yamaha's recommendating the SG300 series but started to occur in some units just a few years ago. Cisco was unable or unwilling to put out any information on a version/revision/lot number, etc., to help identify potential problem units but they did accept the problem units back and performed some type of update or service that corrected the problem. This was done by Cisco on a unit by unit basis.
I do not know if this glitch also affected Dante traffic. That said, I have not had an SG300 that has exhibited this problem.
Lee
The issues with the SG-300 and Q-lan were limited to the Rev B units, which occurred at temperature extremes.
-
The issues with the SG-300 and Q-lan were limited to the Rev B units, which occurred at temperature extremes.
Thanks David!
I could not get that info out of folks at Cisco or QSC.
Lee
-
I've tried troubleshooting this with Audinate multiple times but we haven't been able to figure out what's going on so any help would be greatly appreciated!
I’ve got a Cisco SG300 (Audinate’s recommended switch) and I went through all the settings per the Yamaha manual so it’s setup properly (Been over all kinds of other settings beyond Yamaha’s manual with Audinate).
Running 2 Yamaha CL5’s, 2 RIO’s, 3 ULXD units, X32 rack, and some computers with DVS. Everything is updated to latest Dante firmware, latest DVS software, etc.
- Basically the problem is that the computers on the network will get glitches of high latency (above 20ms) but it’s only for a split second, and it only happens every once in a while (very random). And depending on how much data is being received by the computer depends on how much of a latency glitch there will be. (Only 5 multicast flows to a computer - it gets about 6-8 msec of latency; 20 multicast flows - 20+ msec of latency).
When multi tracking and it gets a latency “glitch" - it causes the computer to lose sync (even though Dante doesn’t show any sync errors or packet errors). When it loses sync, the multitrack also glitches pretty bad till it sync's up again (only a few seconds typically). Occasionally a computer will even act as if it was kicked off the Dante network (it will still receive flows from all the other computers on the network - but it stops receiving flows from Consoles / RIO’s / ULXDs). DVS shows it's still working though and there's no errors anywhere.
I've swapped all ethernet cables to computers, RIOs, CL5's. Purchased another Cisco SG300 switch. Everything is on a redundant network and both switches are the same Cisco SG300-28.
Any help would be greatly appreciated!
Have you tried a different computer? Some computers just don't get along with DVS. My x230's work great, about 75% of the time. The other 25% of the time I need to reboot (after a fresh boot) just to get clear audio. That said my mac mini has been flawless for years.
You didn't say what type of computer you are using...but there is always this...
Tune Windows PC for Best Audio Performance (https://www.audinate.com/faq/how-can-i-tune-windows-pc-best-audio-performance)
-
And have you tried to isolate the issue by removing devices one after eachother?
If dvs is on mac you need to check your network settings.
-
That is one thing I haven’t tried to do but I probably should and see if I come up with anything. To answer some of the other questions -
When I was troubleshooting with Audinate, they suggested putting one CL5 as preferred Master and Sync to External and set that CL5 to internal sync - not Dante. Hasn’t helped.
And to answer David - I’ve tried multitracking on at least 5 different computers (Mac Mini, IMac, MacBook Pro - all fully upgraded new machines) and have the same issues on all of them. Which makes me think it can’t be a computer thing.
What network settings (since all we use are Macs) are you referring to?
And have you tried to isolate the issue by removing devices one after eachother?
If dvs is on mac you need to check your network settings.
-
That is one thing I haven’t tried to do but I probably should and see if I come up with anything. To answer some of the other questions -
When I was troubleshooting with Audinate, they suggested putting one CL5 as preferred Master and Sync to External and set that CL5 to internal sync - not Dante. Hasn’t helped.
And to answer David - I’ve tried multitracking on at least 5 different computers (Mac Mini, IMac, MacBook Pro - all fully upgraded new machines) and have the same issues on all of them. Which makes me think it can’t be a computer thing.
What network settings (since all we use are Macs) are you referring to?
One thing that can get to you on any computer is if there is competition for access to the hard drive. Data rates are very fast but the mechanics of disk access are very slow.
In general it is risky to write audio to the system disk. That is, the boot disk most of the time. Use a second disk or a solid state disk.
The problem is that the operating system will decide to do some disk access at a random time which will cause some audio data loss.
This may not be your problem but I have seen it plenty in my years.
Sent from my iPad using Tapatalk Pro
-
http://forums.prosoundweb.com/index.php/topic,166621.msg1536866.html#msg1536866
-
I will for sure try this as I know this to be an issue as well... it's just that when I monitor HD performance and what not it's barely working (4% or less). And I guess I just don't know how this could effect network latency.
And Robert - I looked through that forum - checked the ethernet settings and they are all just full-duplex on the computers. One interesting thing to note is when I change the setting to "full duplex, flow-control" it instantly boots the computer off the network... doesn't even act like the ethernet is plugged in. (this can't be normal)
I think the first thing I'll try is removing things one by one off the network and see if anything is affected. That seems to be about the only thing I haven't tried.
One thing I did that actually "works" is I moved the whole network to Daisy Chain and plugged a Mac into the Secondary port on the CL5 and multitracked that way and it worked perfect... latency was very consistent and around .500 msec or less. (This was something audinate suggested to try). But they didn't have an explanation for why this would work and plugging into the switch doesn't. I just prefer to have everything running redundant personally so I don't want to Daisy Chain everything permanently.
One thing that can get to you on any computer is if there is competition for access to the hard drive. Data rates are very fast but the mechanics of disk access are very slow.
In general it is risky to write audio to the system disk. That is, the boot disk most of the time. Use a second disk or a solid state disk.
The problem is that the operating system will decide to do some disk access at a random time which will cause some audio data loss.
This may not be your problem but I have seen it plenty in my years.
Sent from my iPad using Tapatalk Pro
-
I haven't been able to get a glitch-free DVS recording in a couple of years. Due to other issues I've seen with Dante over the years, I'm very much starting to suspect the (rental) Cisco switches are the cause of the problem. Most likely mis-configuration, but possibly hardware issues as well. Thinking back a few years ago, I had a different set of switches in my Yamaha system, and had no problems with multitracking. Too bad the Yamaha switches are so damned expensive!
Is there a "settings file" that can be uploaded to a Cisco SG300 series switch? I'm thinking like:
- Factory reset switch to defaults and as new settings
- Load a settings file into the switch with "Known best Dante settings" which comes from Audinate or Yamaha
From what I can tell, there's so many menus and settings in the Cisco, it would be easy for something to be set wrong (especially for a non-IT person) and go unnoticed...
If Audinate or Yamaha won't put out a settings file (does Cisco have a way to set their switches from an external file?) then maybe the community here can put one together?
-
I haven't been able to get a glitch-free DVS recording in a couple of years. Due to other issues I've seen with Dante over the years, I'm very much starting to suspect the (rental) Cisco switches are the cause of the problem. Most likely mis-configuration, but possibly hardware issues as well. Thinking back a few years ago, I had a different set of switches in my Yamaha system, and had no problems with multitracking. Too bad the Yamaha switches are so damned expensive!
Is there a "settings file" that can be uploaded to a Cisco SG300 series switch? I'm thinking like:
- Factory reset switch to defaults and as new settings
- Load a settings file into the switch with "Known best Dante settings" which comes from Audinate or Yamaha
From what I can tell, there's so many menus and settings in the Cisco, it would be easy for something to be set wrong (especially for a non-IT person) and go unnoticed...
If Audinate or Yamaha won't put out a settings file (does Cisco have a way to set their switches from an external file?) then maybe the community here can put one together?
I have been using this for years.
Yamaha SG-300 Setup Guide (http://www.yamahaproaudio.com/global/en/training_support/selftraining/dante_guide/chapter3/01_setting_sg300/)
You could perform the setup once, then save those settings (last page of the linked setup guide). The problem is that when you load those settings into a new switch, unless I am mistaken, all switches will have the same IP address and name. Which could get a little messy if you try to program it while connected to the network.
-
I could see how it would be a Cisco switch problem - but I’ve swapped the switch out with a TP Link switch as well as Meraki switches and everything acts the same. I’ve also setup the Cisco using that Yamaha guide.
I have been using this for years.
Yamaha SG-300 Setup Guide (http://www.yamahaproaudio.com/global/en/training_support/selftraining/dante_guide/chapter3/01_setting_sg300/)
You could perform the setup once, then save those settings (last page of the linked setup guide). The problem is that when you load those settings into a new switch, unless I am mistaken, all switches will have the same IP address and name. Which could get a little messy if you try to program it while connected to the network.
-
I could see how it would be a Cisco switch problem - but I’ve swapped the switch out with a TP Link switch as well as Meraki switches and everything acts the same. I’ve also setup the Cisco using that Yamaha guide.
But.... You did say it works properly when the switches are not used, correct? Sounds like you at least know now where the problem lies - in the switches... Just reinforces what I was suspecting before. I would like to compare to a Yamaha switch if I could ever get my hands on 4 of them!
-
Yea I guess the problem lies in the switches... but why? Seems like others have had this issue as well... but then is it really the switches? Just seems off... I'll rent a few of the yamaha's for a week and see what happens.
But.... You did say it works properly when the switches are not used, correct? Sounds like you at least know now where the problem lies - in the switches... Just reinforces what I was suspecting before. I would like to compare to a Yamaha switch if I could ever get my hands on 4 of them!
-
Yea I guess the problem lies in the switches... but why? Seems like others have had this issue as well... but then is it really the switches? Just seems off... I'll rent a few of the yamaha's for a week and see what happens.
I'd be very curious to know. Would love to find out if all my troubles were due to the Cisco switches.
BTW - you might want to fix your username, even if your name is Derek Toews, you might want to separate them First [space] Last as per the rules here...
-
Yea I guess the problem lies in the switches... but why? Seems like others have had this issue as well... but then is it really the switches? Just seems off... I'll rent a few of the yamaha's for a week and see what happens.
I've used a wide variety of Cisco switch combinations with Dante over the last 2-3 years as well as using good ol' dumb Netgear Gigabit switches far more times than i care to admit. The only switch combo that has NEVER given me any issues is the dumb Blue Netgear switches.
Am i recommending those for your situation? No. I recognize ALL the potential issues there. But it's interesting to me that the most reliable consistent performance i've gotten with a wide variety of Dante applications is the most ubiquitous, cheap generic dumb switch available....
-
Thanks - I'll update the name.
I use the Netgear $30 switches at a few places that have smaller Dante networks and never have had an issue either but I haven't tried it with more than 5 things on Dante.
I'd be very curious to know. Would love to find out if all my troubles were due to the Cisco switches.
BTW - you might want to fix your username, even if your name is Derek Toews, you might want to separate them First [space] Last as per the rules here...
-
So random question to maybe help me troubleshoot this some more.
What is the average latency on the computers on your Dante network?
Since I know the latency is an issue, maybe I can figure it out better and know what the latency is "supposed to be".
I've used a wide variety of Cisco switch combinations with Dante over the last 2-3 years as well as using good ol' dumb Netgear Gigabit switches far more times than i care to admit. The only switch combo that has NEVER given me any issues is the dumb Blue Netgear switches.
Am i recommending those for your situation? No. I recognize ALL the potential issues there. But it's interesting to me that the most reliable consistent performance i've gotten with a wide variety of Dante applications is the most ubiquitous, cheap generic dumb switch available....
-
So random question to maybe help me troubleshoot this some more.
What is the average latency on the computers on your Dante network?
Since I know the latency is an issue, maybe I can figure it out better and know what the latency is "supposed to be".
Super high quality question, but one i can't answer since 99 percent of my use of Dante is in the field on shows, so i can't really go back and check. The other caveat to my uses is that the network is a different combination of devices and switches and and and every time. so a lot of variables...
That being said, i've got an install i'm currently working on where i CAN check just this sort of thing. It's also on a Cisco Meraki switch which means i can also look at things like how much traffic is actually being generated by all my various Dante devices as well as other nifty things.
Stay tuned. I'll do some testing in the next couple days and hopefully share something useful...
-
That'd be awesome. Thank you!
Super high quality question, but one i can't answer since 99 percent of my use of Dante is in the field on shows, so i can't really go back and check. The other caveat to my uses is that the network is a different combination of devices and switches and and and every time. so a lot of variables...
That being said, i've got an install i'm currently working on where i CAN check just this sort of thing. It's also on a Cisco Meraki switch which means i can also look at things like how much traffic is actually being generated by all my various Dante devices as well as other nifty things.
Stay tuned. I'll do some testing in the next couple days and hopefully share something useful...
-
So random question to maybe help me troubleshoot this some more.
What is the average latency on the computers on your Dante network?
Since I know the latency is an issue, maybe I can figure it out better and know what the latency is "supposed to be".
I just set it at the highest possible value. 10ms? 20ms?
Don’t care about latency as it’s just a recording.
I do though care a lot about glitches and skips, which is what I’ve been getting.
Sent from my iPhone using Tapatalk Pro
-
I've been having some issues lately with Dante on my SG300's as well. we've added more edge devices and sometime the extreme ends of the network are not coming up, or show in DC but produce no audio when patched. odd as you can ping all of the units but they still don't work properly.
we narrowed the issue down to our SG300's, and have found success by removing the multicast and IGMP settings as recommended by the Yahama guide. all other settings from the guide remain. haven't gone further than making it work, hopefully have time in the next month or so to see just why this is causing us grief. thinking maybe IGMP should only be on in one switch, we have as many as 12 switches in play end to end. going without IGMP at all for now, no issues so far.
for the record, our network for Dante is its own VLAN with no other traffic. this VLAN (and others) trunk between switches. we use Cisco SG300, TP-Link managed and unmanaged switches across the network.