Abuse of statistics makes legitimate engineering use harder

One the easiest ways for a lazy or ignorant journalist (or reporter) to crank out a story is to grab a few numbers, do a fast and loose analysis, and then extrapolate a trend. Alternatively, the journalist can cite a few data points (even a single one will do, in a pinch) to give the veneer of substance and support to whatever is on his or her mind. Even the venerable New York Times, which likes to think of itself as the “newspaper of record” (whatever that actually means these days) has fallen for this trap repeatedly, and it's gotten more frequent and obvious in the past few years

A recent article (“Online sales lose steam,” June 16, 2007) hypothesized that buying on the Internet was losing appeal to customers, because the annual growth had slowed from around 25% to about 10%. I'm not arguing with those raw data points, but with the interpretation. Here's why: first, if overall retail sales in general are growing around 5%, as they are, then 10% growth is still much, much better than the market as a whole and quite impressive. Second, the math of exponential growth and the law of large numbers demonstrate that you can't exceed the growth of the base number by a large factor for too long, or pretty soon you'll have taken over the whole economy, which can't happen.

But the authors of this story didn't bother to get reality get in the way of their premise, and while that's sad, such statistical abuse is becoming more common. I laugh when analysts point to some market for a class of ICs or application which is growing rapidly from a very small base, and then imply strongly that this sort of grown can continue for as far they can see. Will the market for cell phones, DVRs, video games, Apple iPhones (you can cite whatever is hot this year, it doesn't matter) grow at 20% annually for the foreseeable future until everyone on the planet owns several of each? I don't think so.

Why do I care? As an engineer, I know that good design is a combination of data, analysis, and intuition in the right balance. Sometimes the numbers support the intuition, sometimes they do not, but at least you have to be honest about what the numbers say. Maybe the data is right, but the setup is wrong. Or maybe the data has inherent margins of error that mean that a wide range of interpretations is possible. Or maybe you are confusing a slowdown in growth with an actual slowdown.

I am always fascinated by those high-priced market research reports which predict the market for something will grow by 57.34% in the next five years; heck, that's a crystal ball with four significant figures. In fact, it would be more meaningful to say the market will grow by 57% with a error band of around 10 or 20% around it. The accuracy implied by the false precision is hilarious and sad at the same time.

Numbers are great, numbers are important, but their misuse either through ignorance, laziness, or both by so many makes it hard for engineers. We know that we have to use them, but use them with caution.

0 comments on “Abuse of statistics makes legitimate engineering use harder

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.