ProSoundWeb Community

Sound Reinforcement - Forums for Live Sound Professionals - Your Displayed Name Must Be Your Real Full Name To Post In The Live Sound Forums => LAB: The Classic Live Audio Board => Topic started by: Pat Semeraro on September 02, 2018, 03:40:12 PM

Title: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 02, 2018, 03:40:12 PM
Good day friends!

It seems the loudspeaker info I'm always looking for is almost never shared by manufacturers.  Let's take subs for instance.  Every manufacturer has a parking lot and every manufacturer should have a copy of SMAART.  It's easy to buy/rent your competitors products.

While the set below doesn't tell you everything... it would be a start.

For passive boxes, set up in a common spot and measured at a common spot, using a common amp (pick one that can do the job and use the same amp on every test) and using the box manufacturers dsp, log the following data and put in a chart to compare against the other boxes.

1) 20hz - 200hz sweep at full power (per box dsp) with spl on top, phase trace and distortion shown underneath.
2) transfer function of an 808 kick drum (software generated is fine just use the same one on every test) at full power (per box dsp) with spl on top and distortion shown underneath.  I'm particularly interested in transient compression and response re-shaping in limiting.
3) waterfall plot from single cycle tone bursts with spl shown. 
4) 30 hz sinewave at full power (per box dsp) for 5 minutes with spl and distortion shown over time.


Powered boxes would get the same test but obviously the amp/dsp is included.  Logging the drive voltage might be useful to some folks.

I would think manufacturers would be eager to show how their boxes outperform their competitors in those four benchmarks.  Without that data, how can I meaningfully compare one sub against another? 

Pat Semeraro
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Art Welter on September 02, 2018, 04:09:22 PM
I would think manufacturers would be eager to show how their boxes outperform their competitors in those four benchmarks.  Without that data, how can I meaningfully compare one sub against another? 
Pat,

I can't think of any manufacturers eager to show 2 and 3 digit distortion results your tests would reveal.

To meaningfully compare subs (or any loudspeaker) without requiring manufacturers to adhere to a set of 4 benchmarks that virtually none would agree to, will still require your own evaluation and comparisons.

Art
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Rob Spence on September 02, 2018, 05:25:54 PM
Why would they? What is in it for them?

Likely they would only compare to ones they are much better than vs the ones YOU want to see. They certainly don’t want to compare to better speakers.

It is in their best interests (they think) to get you to buy based on marketing hype since you are not likely to do serious measuring after buying.


Sent from my iPad using Tapatalk Pro
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 02, 2018, 06:50:31 PM
Pat,

I can't think of any manufacturers eager to show 2 and 3 digit distortion results your tests would reveal.

To meaningfully compare subs (or any loudspeaker) without requiring manufacturers to adhere to a set of 4 benchmarks that virtually none would agree to, will still require your own evaluation and comparisons.

Art

Art, you and I agree that virtually no one will agree on specific measurements and methods.

What I'm suggesting is the manufacturer choose a set of measurements that reflect real world usage and test everything (their boxes and competitors boxes) exactly the same way.  Evaluating loudspeakers is not the same thing as wine tasting.  In a world where watts (voltage...) are plentiful and cheap and dsp is endlessly powerful, I'm interested in the actual in-situ performance and how it compares to other available options.   

That should be on the manufacturers to demonstrate their products performance, not just us as users.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Ivan Beaver on September 02, 2018, 08:25:49 PM
You do realize that you are asking somebody to rent thousands of boxes and do those tests----------

There are so many factors involved in trying to do this.

None of those test would tell how the box "sounds".

Your tests could be very destructive and actually tear up the loudspeakers.

While it "sounds" like a good idea, there are many "holes" in the idea.

What is "acceptable" to some people is not acceptable to others, they want some different tests.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 02, 2018, 09:54:06 PM
You do realize that you are asking somebody to rent thousands of boxes and do those tests----------

There are so many factors involved in trying to do this.

None of those test would tell how the box "sounds".

Your tests could be very destructive and actually tear up the loudspeakers.

While it "sounds" like a good idea, there are many "holes" in the idea.

What is "acceptable" to some people is not acceptable to others, they want some different tests.

Ivan, thanks for jumping in!  I agree there are many holes, its an idea, not a ready to implement plan.

I don't think "thousands" of boxes are necessary to offer meaningful data to the market.  A manufacturer knows the top few direct competitors to a given product, and testing those would be a great start.

As for tearing up the boxes, I don't see how.  The dsp should account for amp gain and manage both driver excursion and long term thermal per the manufacturers configuration or via a closed loop system.  Any system that doesn't do that certainly cannot be "full power" tested. (Who would even know what "full power" means in that case?) Apologies if I was not clear about that and looking at an actual methodology for testing would probably require the manufacturer's matching parts.  (i.e. Crown amps/dsp for JBL speakers, Powersoft amps/dsp for EAW, LA-RAK for L'Acoustics, etc.)  Powered boxes obviously make this a moot point.

As for acceptability, I don't know how to answer that.  Its difficult to imagine that shoppers and buyers would find it unacceptable to see a credible overlay graph of five competing subs (processed/powered per the manufacturers configuration) measured at the same time in the same space, for response, efficiency, time domain performance, distortion, and maximum manufacturer configured output.

Does this offer any clarity to my original post?

Pat Semeraro

Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Tim McCulloch on September 03, 2018, 04:47:44 AM
Ivan, thanks for jumping in!  I agree there are many holes, its an idea, not a ready to implement plan.

I don't think "thousands" of boxes are necessary to offer meaningful data to the market.  A manufacturer knows the top few direct competitors to a given product, and testing those would be a great start.

As for tearing up the boxes, I don't see how.  The dsp should account for amp gain and manage both driver excursion and long term thermal per the manufacturers configuration or via a closed loop system.  Any system that doesn't do that certainly cannot be "full power" tested. (Who would even know what "full power" means in that case?) Apologies if I was not clear about that and looking at an actual methodology for testing would probably require the manufacturer's matching parts.  (i.e. Crown amps/dsp for JBL speakers, Powersoft amps/dsp for EAW, LA-RAK for L'Acoustics, etc.)  Powered boxes obviously make this a moot point.

As for acceptability, I don't know how to answer that.  Its difficult to imagine that shoppers and buyers would find it unacceptable to see a credible overlay graph of five competing subs (processed/powered per the manufacturers configuration) measured at the same time in the same space, for response, efficiency, time domain performance, distortion, and maximum manufacturer configured output.

Does this offer any clarity to my original post?

Pat Semeraro

Hi Pat-

Both Ivan and JR have posted about "lies, damn lies, and specifications" (er... statistics, according to Mark Twain) and for various reasons it all comes back around to one of Ivan's lines:  "the great thing about standards is there are so many to choose from."

As was mentioned up-thread the reason the "big boys" don't usually play along in such things is they have nothing to gain from helping in such comparisons; it's the big boys that everyone else wants to be as good as, or better than, in some measurement contest.  We make a mistake in presuming that all measurement is equal and consistent when we don't know how it was done, and when we do the Measurement Patrol start swooping down.

If you'd like to see the results of the Monday-morning Measurement Quarterbacking you can search the LAB archives for the NYC Subwoofer shootout.  A bunch of subs, a less than ideal measurement space, some transducer powering inconsistencies... and every measurement becomes suspect in someone elses eyes.  No deception, but not possible to guaranty that all measurement circumstances were identical.

None of the TLA brands sent subs but there was a decent selection of products and is how I became acquainted with David Lee of BassBoss and Jeff Permanian of JTR Loudspeakers... anyway, the adoring public decided the TEF sweeps couldn't be reliably compared.  The point of this little memory lane trip is to reinforce that making reproducible measurements is not a trivial task and requires specific testing protocols and lots of space.

So every time a product is erroneously measured, or measured differently than another product, whatever difference becomes marketable.  "Now with 3% more even-harmonic distortion for a warmer sound!" What it won't do is tell a prospective buyer what measured differences, if any, will actually sound like.

And that's what the NYC Sub shootout was about - how these 20-odd subs sounded in the real world of a Manhattan night club, and whatever else that might be extrapolated from listening and making the TEF plots (whatever ones impression of reliability).

And that circles us back to "numbers can't tell us what something *sounds* like".
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Lyle Williams on September 03, 2018, 05:50:25 AM
Nobody's box will take 30Hz at full power for 5 minutes.

That'd be a real problem, except the "30Hz hum for five minutes" genre has gone out of fashion in most venues.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Rick Powell on September 03, 2018, 05:58:37 PM
Josh Ricci’s Data-Bass site attempts to do direct measurement comparisons for dozens of commercially available and custom-design subs, but most brands of SR subs have yet to be tested there. It caters a little more to the home theater crowd.

I think a site like this could provide a “Consumer Reports” type evaluation independent of market claims, but for such a limited consumer market for pro audio, not sure if it would be feasible.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Rick Alan on September 03, 2018, 06:04:02 PM
Because all the numbers are made up. 

The only way to compare is with your ears.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: John Roberts {JR} on September 03, 2018, 06:14:06 PM
Do not expect easy answers to complex questions. We have discussed this at length right here as long as I can remember here being here.

This is not some scam or conspiracy, it's just complicated and few customers are willing to do the work to understand the complicated answers so will always be led around by the nose chasing "simple" but wrong answers to complicated questions.

JR
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 03, 2018, 08:36:28 PM
Why would they? What is in it for them?


Rob, I would think everything.  Showing that your box is cleaner, more efficient, has more maximum output, (or is smaller with same output) is more dynamically accurate, etc. than your competitors would be powerful information for manufacturers to share with prospective buyers.

Pat
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Dave Garoutte on September 03, 2018, 08:38:42 PM
Marketing is not about the truth.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: John Roberts {JR} on September 03, 2018, 09:07:12 PM
never mind

JR
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 03, 2018, 09:17:39 PM
Hi Pat-

And that circles us back to "numbers can't tell us what something *sounds* like".

Tim, you and I (mostly) agree on your points and that's why I suggest the manufacturer do the measurements. If we trust a manufacturer to build the tools we use to earn a living, we should be able to trust them to tell us how those tools work.   

I was also specific about measuring outside, as measuring inside is often problematic (expecially at very low frequencies) unless you're in a chamber.  And measuring in a chamber can tell us some useful stuff but will not tell us how that sub sounds in a parking lot.  The same way that measuring a car on a dyno will not tell us how fast it "feels" or how fast it actually goes, but does give useful data to back up why the car goes or feels that way it does.

Part of my issue is the numbers that manufacturers share.  A phase trace, a transfer function measurement of a dynamic signal, (I suggested the most famous kick drum in history) and a waterfall plot can tell us a lot about how a sub will "sound" and tell us even more about how a sub sounds different at different power levels. 5 minutes of 30Hz tells us much more about power handling than the fake number on the spec sheet. Instead, we get specs like "calculated maximum output" and "6000 watts max power handling" that tell us little about what a sub can do in use.

As customers, I guess we've become ok with that because certain products "work well" or "sound good" or "everybody uses them", etc.      We're relying on opinions instead of data.

I'm not trying to fulfill a rider for this particular project and its not like I can go to the "sub store" and compare the top 6 touring subs with the top 6 alts subs to even form an "opinion" about what "I think" "sounds good" or be able to understand size vs cost vs output vs durability in an empirically meaningful way.

Thanks for your thoughtful response.

Pat
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 03, 2018, 09:19:57 PM
Marketing is not about the truth.

Dave, that sounds cynical.  I've posed what I believe is a serious question and am not sure how your response helps move my understanding forward?

Pat
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: David Sturzenbecher on September 03, 2018, 09:20:12 PM
Rob, I would think everything.  Showing that your box is cleaner, more efficient, has more maximum output, (or is smaller with same output) is more dynamically accurate, etc. than your competitors would be powerful information for manufacturers to share with prospective buyers.

Pat

[scenario]
I can prove my box is cleaner, but the other guys box is more efficient, and someone elses box box has more max output.

I dont think I want to post results that show someone elses box beat me in two categories.
[/scenario]

You forgot to mention cheaper.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 03, 2018, 09:26:12 PM
Because all the numbers are made up. 

The only way to compare is with your ears.

Rick, I'm sure you're a nice guy and maybe I've just misunderstood the meaning of your post.
Your response sounds more like a strong opinion than a reasonable statement of fact.  I'm trying to answer a serious question and there are people here that I respect and trust offering thoughtful responses. 

Pat

Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 03, 2018, 09:42:42 PM
[scenario]
I can prove my box is cleaner, but the other guys box is more efficient, and someone elses box box has more max output.

I dont think I want to post results that show someone elses box beat me in two categories.
[/scenario]

You forgot to mention cheaper.

David, I get it... but I'm not looking for the cheapest box.  Those shoppers have different requirements.  I'm struggling to find some way of doing a cost/benefit analysis that uses real data. 
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: David Sturzenbecher on September 03, 2018, 09:44:50 PM
David, I get it... but I'm not looking for the cheapest box.  Those shoppers have different requirements.  I'm struggling to find some way of doing a cost/benefit analysis that uses real data.

I guess what is keeping you from renting in the boxes you are interested in, and taking the measurements that are relevant to you?
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Russell Ault on September 04, 2018, 12:29:57 AM
I sadly can't remember who I was talking with, but I remember someone working for Meyer mentioning that part of the reason they created MAPP was to avoid publishing written (and therefore easy to misinterpret) specifications of their speakers. Their philosophy is that if you need more information than they publish in print, you can run a test scenario in MAPP and get way more (useful) information than you'd get off of a sheet of paper. Obviously it won't quite do the tests you're trying to do, but it will start heading in the direction you're looking for. (I'm not as familiar with, say, ArrayCalc or Soundvision, but I'm guessing they'll do something similar for you, too.)

Those shoppers have different requirements.  I'm struggling to find some way of doing a cost/benefit analysis that uses real data. 

Cost vs. what benefit? To my mind all shoppers have different requirements: system interfacing, inventory commonality, rigging capabilities, arraying convenience, truck pack, rider acceptability, processor compatibility, even physical appearance are all "benefits" that can be more important than performance. Even within performance, what's important for one application may be meaningless in others (e.g. I've used some good-sounding, high-quality subwoofers that would choke if you fed them a strong 30Hz signal, but that's okay because those shows had no useful content below 40Hz anyway).

-Russ
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Tim McCulloch on September 04, 2018, 02:05:32 AM
I sadly can't remember who I was talking with, but I remember someone working for Meyer mentioning that part of the reason they created MAPP was to avoid publishing written (and therefore easy to misinterpret) specifications of their speakers. Their philosophy is that if you need more information than they publish in print, you can run a test scenario in MAPP and get way more (useful) information than you'd get off of a sheet of paper. Obviously it won't quite do the tests you're trying to do, but it will start heading in the direction you're looking for. (I'm not as familiar with, say, ArrayCalc or Soundvision, but I'm guessing they'll do something similar for you, too.)

Cost vs. what benefit? To my mind all shoppers have different requirements: system interfacing, inventory commonality, rigging capabilities, arraying convenience, truck pack, rider acceptability, processor compatibility, even physical appearance are all "benefits" that can be more important than performance. Even within performance, what's important for one application may be meaningless in others (e.g. I've used some good-sounding, high-quality subwoofers that would choke if you fed them a strong 30Hz signal, but that's okay because those shows had no useful content below 40Hz anyway).

-Russ

The Meyer thing makes sense; it's better to show someone a results picture than to have them incorrectly interpret any number of specifications, charts and graphs.

I understand that what Pat is hoping for is a multi-spec measurement done in the same manner so results can be compared.  The questions are "what are we then comparing" and "can we interpolate these numbers, graphs and charts into some from of auralization?"  As I pointed out earlier, differences in some criteria (regardless of importance) will become marketing fodder; "1000W with Zero Transfat!".  8)

The industry has been successful in measuring electrical performance, efficiency, and transfer function for some time, but among units that measure very similarly they still sound different... and that's why I suspect this is a solution looking for a problem, and the solution itself will present its own new gamesmanship and marketing-speak/spew.

The OCD side of me likes what Pat is aspiring for but the practical side of me isn't too enthused.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: TJ (Tom) Cornish on September 04, 2018, 09:49:02 AM
David, I get it... but I'm not looking for the cheapest box.  Those shoppers have different requirements.  I'm struggling to find some way of doing a cost/benefit analysis that uses real data.
Pat, I think most of us sympathize with you, but what you want isn't going to happen.  Sound quality is extremely subjective and the tonality of a box is about the fourth-most important factor in what the punters hear - stage talent is most important, room acoustics and stage wash usually fight for spots 2 and 3.  Arguably pattern control is more important than sound quality, which puts what you are trying to measure maybe at number 5.

Back to the business side of things, how easy is the rigging?  How heavy are the boxes?  How many people does it take to deploy the system?  How are the carts/trunks/cases designed?  Is it a good truck pack?  How many amp channels and how much copper is required to make noise?  Does the company have good support?  What does the rider say about these boxes?

These are the questions that people in business tend to focus on. 
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Josh Ricci on September 04, 2018, 12:05:37 PM
Josh Ricci’s Data-Bass site attempts to do direct measurement comparisons for dozens of commercially available and custom-design subs, but most brands of SR subs have yet to be tested there. It caters a little more to the home theater crowd.

I think a site like this could provide a “Consumer Reports” type evaluation independent of market claims, but for such a limited consumer market for pro audio, not sure if it would be feasible.

That's sort of what I'm going for is exactly the consumer reports type of information. I'd say it leans a bit more DIY than anything. I want to get more products from the SR arena but the few inquiries I've made have not gotten a response from the MFG's. They do not seem to be interested. It's unsurprising really...That whole "how does it benefit us" business.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Dave Garoutte on September 04, 2018, 12:23:27 PM
That's sort of what I'm going for is exactly the consumer reports type of information. I'd say it leans a bit more DIY than anything. I want to get more products from the SR arena but the few inquiries I've made have not gotten a response from the MFG's. They do not seem to be interested. It's unsurprising really...That whole "how does it benefit us" business.

Because it is a BUSINESS.  Why would they want to provide information that could be used against them?  They are going to plead the 5th.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Rory Buszka on September 04, 2018, 12:41:36 PM
Because it is a BUSINESS.  Why would they want to provide information that could be used against them?  They are going to plead the 5th.

And besides this, what assurance would an independent customer have that the data was responsibly collected and the analysis was not slanted somehow to favor the manufacturer? Could you ever be certain? Even when some third-party standard is in place, such as CEA2010 distortion thresholds, independent practitioners have observed differences in their results from one data source to the next, which seem to be due to lurking variables in methodology and hardware. As far as I know these differences remain unresolved, and naturally each practitioner says "just compare measurements I made to other measurements I made", so CEA2010 distortion measurements are, for now, still not truly universal. If consultants and measurement specialists are unable to agree, when they are striving to add value to the standard by harmonizing their methodology, what will happen when every manufacturer is taking data separately? What if manufacturer A's measurements of third party C's subwoofer show meaningful differences from manufacturer B's measurements of third party C's subwoofer, and to make things more interesting, what if both A's and B's measurements differ from the published measurements by third party C of their own subwoofer? Who is right?

Something like what the OP describes was done by Community Loudspeakers back in its early days for off-axis measurements of speaker horns, and they did publish a book called the White Book containing their results, but I don't know if any copies of it are surviving and I've never seen one. But their White Book was the product of their efforts alone, and not a cooperation between manufacturers.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Weogo Reed on September 04, 2018, 11:49:08 PM
Hi Pat,

One place for some independent testing, with pretty good apples to apples comparisons:
 
 http://clfgroup.org/

Mostly install boxes, plus a few seen out in the gigging world.

Good health,  Weogo
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 05, 2018, 12:47:59 AM
Good day friends!


Thank you everyone for so many thoughtful responses!  The hope was that maybe I've been hiding under a rock and just didn't know where to look for more comprehensive, closer to the real use data.

Over ...a few...   decades I've used an awful lot of subs from junky things I built myself in the 80s, to more recently lots of Meyer 1100s in stadiums and dozens and dozens of others in the years between.  Seen scoops and folded horns go in and out of style, heard most of the band-pass and trick subs at the MI level, used many EAW both the older RCF loaded boxes and the later 18 sound/B&C loaded versions.  Most every box by Meyer, the EV 4-leaf clover sub, four generations of JBL 18s in various boxes (2440, then, 2441, then 2442, then current neo 18s)  I'll stop rather than go on and on.

While I have strong opinions of how they all "sound" and anecdotal knowledge of what different boxes do well, or adequate, or poorly, (in my opinion which might differ from yours!) my OCD side still would like to see trustworthy data.

In corporate, where house and rental gear was going in and out constantly, with multiple shows coming in and and out of the shop each week, the logistics of putting 6 different boxes in our parking lot and measuring was basically impossible.  Measuring one box per day and trying to overlay the data later is a recipe for results that are less than useful.

My intrigue is maybe not shared by some simply because many have their rigs and that's what they provide to their market or clients.  My world was very different.  OK with EAW, great.  Has to be Meyer?  Fine.  Prefer L'Acoustic but you can live with the vertec that's available for your show date?  Check.  Has to be the purple brand?  Nope, call somebody else...

The result of all that is I never invested emotionally in a brand.  (I do have strong opinions but does that matter?) Without an emotional component to push me or a rider to fill, I'm left with only my opinion and empirical data and trying to compare  data is where I got stuck. 

My current project has space for one pair of subs and having recently sold my business, I no longer have access to a shop full of stuff to play with.  Knowing how a dozen JBL 728s sound in a ballroom or 18 Meyer 1100s sound in a stadium or 24 SB1000s sound in a field doesn't exactly scale down to this application. 

Again, appreciate all the good comments!
Cheers, Pat
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 05, 2018, 12:52:29 AM
Hi Pat,

One place for some independent testing, with pretty good apples to apples comparisons:
 
 http://clfgroup.org/

Mostly install boxes, plus a few seen out in the gigging world.

Good health,  Weogo

Interesting, thank you!
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Pat Semeraro on September 05, 2018, 01:03:08 AM

These are the questions that people in business tend to focus on.

TJ, agree with you 1000%.  I'm asking a different, and sort of utopian question to the experts here.  This project will be one pair of subs, sitting on the floor until the end of time.  With most of the practical/transport/rigging stuff off the table, I have the luxury of thinking about sound quality but with the constraint of space.

I'm sure that my suggested testing would show some standout performers, and wish the manufacturers would be more proactive in helping me understand why their particular product is a better option.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Duane Howell on September 05, 2018, 12:25:22 PM
I simply can't imagine any manufacturer consenting to letting another manufacturer publicly and officially publish spec data for their product. It's a huge conflict of interest, it's like having the tobacco industry publish a health study on smoking. I wouldn't be surprised if there are legal issues too.

Your best bet would be some sort of third party database or organization, but I still think it would be rife with politics and conflicts of interest.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Josh Ricci on September 06, 2018, 05:19:30 PM
And besides this, what assurance would an independent customer have that the data was responsibly collected and the analysis was not slanted somehow to favor the manufacturer? Could you ever be certain? Even when some third-party standard is in place, such as CEA2010 distortion thresholds, independent practitioners have observed differences in their results from one data source to the next, which seem to be due to lurking variables in methodology and hardware. As far as I know these differences remain unresolved, and naturally each practitioner says "just compare measurements I made to other measurements I made", so CEA2010 distortion measurements are, for now, still not truly universal. If consultants and measurement specialists are unable to agree, when they are striving to add value to the standard by harmonizing their methodology, what will happen when every manufacturer is taking data separately? What if manufacturer A's measurements of third party C's subwoofer show meaningful differences from manufacturer B's measurements of third party C's subwoofer, and to make things more interesting, what if both A's and B's measurements differ from the published measurements by third party C of their own subwoofer? Who is right?

Something like what the OP describes was done by Community Loudspeakers back in its early days for off-axis measurements of speaker horns, and they did publish a book called the White Book containing their results, but I don't know if any copies of it are surviving and I've never seen one. But their White Book was the product of their efforts alone, and not a cooperation between manufacturers.

It can/should be done by an independent 3rd party in theory but the reality is there are a bunch of reasons it would be very difficult for it to work that way which I will outline below.  Verifiable measured data is very important. It should be completely unbiased in theory. Now if the data is being purposely skewed or completely fabricated to favor a product, that is a possibility, but it would be highly unethical.

As far as differences in methodology, etc...Yes this can account for some differences between data sets. You mentioned CEA-2010. There have been a number of instances over the years with data between multiple parties that didn't agree. The majority of these involved outlier data reported by people who were either very new at it or have a number of question marks in their data sets over the years that indicate that it simply isn't being captured or reported correctly and is of questionable trustworthiness. In those cases it usually gets investigated  and becomes apparent which is the "correct" set of data. This type of thing is very rare though. The MFG internal CEA-2010 data has been spot on in every case I've dealt with. By spot on I mean within +/-1dB generally over the operational bandwidth. You simply will never be able to get tighter agreement than this from something like CEA-2010. The hardware and even software differences don't amount to anything to worry about in my opinion. As long as the setup and methodology is good the results will be the same regardless.

The major cause for measurement variation is environmental and atmospheric conditions. This type of measurement collection is usually done outdoors. The second major factor is manufacturing tolerances in the speaker. Unless 2 testers get the exact same speaker this is going to cause some variation. For this reason I don't consider measured output differences as significant until >1dB. Burst output data like CEA-2010 is useful and it can be repeatable enough to be significant, but I find a lot of other measurements to be just as if not more useful and it's a large collection of different types of measurements that can really paint a picture of the overall performance of a speaker. All of those measurements will be subject to the same variance due to environmental and other factors as well but its the overall broad spectrum of performance differences between speakers that matter not overly exact minutia.

With all of that said what I would like to see in a perfect world is the industry move towards more standards similar to CEA-2010, which would then be adopted over time and used by the MFG's. I'd like to see standards covering: Sensitivity measurements, low frequency extension, frequency response, distortion, directivity and power handling. Everything would be based on measurements the measurements would be shown and there would be a suggested format that these should be reported in. Perhaps as a standardized PDF report since I'm dreaming here.

There are a number of problems with an independent 3rd party doing reports. (I speak from years of experience doing exactly this)
1.) Unless you've done a few of these you cannot fully comprehend the amount of work and time involved.
2.) There's so much product out there! (See above. You'd need a large team of people to truly cover even just the Pro market.)
3.) Its been fun for a while as a hobby/charity but is the whole team working for free indefinitely, year after year? (Doubtful)
4.) Generate income to grow and keep the thing going. How to do that? Crowd funding? Subscription? Advertising? Charge for the work/reviews? All of it? It's doubtful that crowd funding and /or subscription service would make enough revenue to offset the huge amount of time and effort. The pool of people that are interested enough in this type of thing is relatively small even if including Pro, consumer and mobile markets.
5.) If you take advertising or charge for a measurement workup from a speaker MFG it opens the door to the perception of being bought and bias towards them.
6.) If a product performs poorly or simply not quite as good as another the MFG may not supply future product for testing or may choose to not publicly release the data (NDA) or both.
7.) It is best not to rely on MFG's to provide product for the above reasons but purchase of products for testing out of pocket is also unsustainable once you get into products costing thousands of dollars each. Most owners of products balk at providing one for performance evaluation once they realize the lead time, shipping costs and potential for damage.

It makes most sense for the MFG's to do this stuff internally as part of product development. Most would assume that they are doing this but not sharing it publicly. You would be surprised how little some actually do.

Another problem is interpretation of the measurements regardless of who takes them. The average consumer/lay person will have no idea what they are looking at or how to interpret and make a useful comparison in most cases.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: John Roberts {JR} on September 06, 2018, 05:30:30 PM
cough....

You can't handle the truth....

JR
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Chris Grimshaw on September 07, 2018, 03:19:32 AM
I think it's more like "you really don't want to know the truth".

To pick a product, take a look at an EV sub where there's both a passive and active option. Note the wild differences in frequency response - any difference is being EQ'd by the built-in amplifier, and I've seen differences as great as 10dB on their mid-line 18" model.
So there'll be 10dB of EQ boost happening, which is 10x the power at some frequencies compared to others. Either you lose that as headroom, or you go for dynamic EQ, where you've got a cabinet with a frequency response that changes (drastically) with level. Neither of those are great options to work with, which is why I build my own speakers.

Chris
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Len Zenith Jr on September 07, 2018, 03:46:13 AM
I think it's more like "you really don't want to know the truth".

To pick a product, take a look at an EV sub where there's both a passive and active option. Note the wild differences in frequency response - any difference is being EQ'd by the built-in amplifier, and I've seen differences as great as 10dB on their mid-line 18" model.
So there'll be 10dB of EQ boost happening, which is 10x the power at some frequencies compared to others. Either you lose that as headroom, or you go for dynamic EQ, where you've got a cabinet with a frequency response that changes (drastically) with level. Neither of those are great options to work with, which is why I build my own speakers.

Chris

You've seen impedance plots of cabinets then. With built in factory programmed DSP, if you only do your boosts where impedance is at a maximum, lets say 38 ohms, then there is little power added to your voice coil. You wouldn't lose your headroom. I'm sure the designers know how to pick and choose their battles.
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Chris Grimshaw on September 07, 2018, 04:23:13 AM
You've seen impedance plots of cabinets then. With built in factory programmed DSP, if you only do your boosts where impedance is at a maximum, lets say 38 ohms, then there is little power added to your voice coil. You wouldn't lose your headroom. I'm sure the designers know how to pick and choose their battles.

They usually boost around the port tuning frequency, where impedance usually gets below the nominal rating of the cabinet.
Even putting thermal considerations to one side, the amp will still run out of voltage swing.

Chris
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: Marcus Baeumler on September 07, 2018, 07:52:21 AM
 Yes, the never-ending great debate…


I my personal opinion this is based on a few assumptions that I do not agree with and for me are the reasons why this will never happen.


1. It is possible to adequately and completely describe the audio performance, or maybe even more important to many, the audio experience of a loudspeaker system with a simple set of measurements that are also easy to understand.
-> I don´t think so…
Just think of something really simple as nominal dispersion: a simple number will not tell you anything. You will at least need a “dispersion over frequency” graph to really understand what is going on and even that is only half of the story.
Max SPL? There are probably 20 different ways to measure this and all of these different ways represent one part of the description of what somebody would expect as an audible experience when comparing one loudspeaker with another.
Short side story: a very highly regarded magazine here in Germany uses three-dimensional graphs to describe the output performance of amplifiers… Watts/Impedance/Crest factor.
And there are many more.
 
2. You will find two manufacturers that agree on a simple set of measurements that adequately describe the audio performance and audio experience of their loudspeakers.
-> Probably not and the reasons for that start with what I described above.
 
3. Everybody actually understands how these measurements relate to their expectations of the audible performance of a loudspeaker system.
-> Hard to judge objectively but on a broader scale I am not so sure. Nominal dispersion and Damping Factor are just some examples.
 
4. These simple measurement results are actually all that important.
-> Yes to many people but by far not for all. There were many posts above that describe a lot of the benefits that can be much more important when buying a system. I personally know of system sales for literally hundreds of thousands of dollars without a datasheet even being looked at.


Marcus

 
Title: Re: Why Do Most Loudspeaker Manufacturers Not Show Comparison Data?
Post by: John Roberts {JR} on September 07, 2018, 10:04:25 AM
I think it's more like "you really don't want to know the truth".
I am repeating myself (again and again and...). Of course they want some version of the truth but are unwilling to educate themselves to understand how to evaluate multiple specifications and put them in correct context for their application (more is not always better).
Quote
To pick a product, take a look at an EV sub where there's both a passive and active option. Note the wild differences in frequency response - any difference is being EQ'd by the built-in amplifier, and I've seen differences as great as 10dB on their mid-line 18" model.
So there'll be 10dB of EQ boost happening, which is 10x the power at some frequencies compared to others. Either you lose that as headroom, or you go for dynamic EQ, where you've got a cabinet with a frequency response that changes (drastically) with level. Neither of those are great options to work with, which is why I build my own speakers.

Chris
As I have also long said I like to have the professional speaker engineers make the design tradeoff decisions for me after thorough evaluation of ALL variables. I haven't made my own speakers since the 70s. Of course there is nothing wrong with rolling your own if it makes personal sense (it hasn't for me for several decades).
=====

I have wrestled with this for years, professionally from the selling side (I supervised the speaker product manager for a decent sized company). He lost a lot of sleep and hair over trying to give customers what they think they want. This is an old and well discussed topic around here.
 
And one more time for the cheap seats most (all) major loudspeaker manufacturers try to present accurate specifications to satisfy customer's desire for information, but simple answers for complicated questions don't exist, and customers will make their purchase decision (often buying something else) long before you can finish explaining the difficult stuff. 

JR