We live in a world permeated with pundits and predictors, spouting conventional wisdom (CW), often based on just a few points of data. Only problem is that very often, this so-called wisdom is wrong on the facts or the interpretation of them. Often, it's the result of an echo-chamber effort, where A says something, B agrees and amplifies it, A points to B as being in agreement, and so on. In engineering terms, it's a feedback circuit that goes into oscillation and sometimes beyond control.
Some organizations are very good at assessing this CW. The Wall Street Journal, for example, has a regular columnist called “The Numbers Guy” who digs deep to see where often-cited data really comes from. His research showed, for example, that the weekend movie box-office figures, touted on Monday as quantifying how well (or not) movies did, are derived from actual and rough-guess sales figures, to which a whole list of correction factors are then applied, based on historical and geographical factors. When all is done, this supposedly firm number can have an error band of up to 30 per cent either way for a given movie—not very good at all!
Similarly, another recent column took a hard look at the data used to support the “long-tail” premise, the focus of a recent best seller by Chris Anderson. In short: most of the actual numbers from sources such as Amazon and Netflix don't support the premise.
What's the connection between CW and engineering design and development? It's relevant in two ways. First, when you are deep in design and debug, and you and your team start making assumptions and hypothesis about what the problem may be, be clear to yourself and your managers that these are not demonstrated facts, yet. There's nothing wrong with proceeding based on assumptions, in fact, that's the basis for many successful engineering efforts—but be honest with yourself as to what's truly known and what is maybe known. Luckily, for many engineering hypothesis, we can collect good data, run tests, do A/B comparisons, and more, so we can develop solid data and answers,
Second, when those marketing folks start telling you what's going to happen, ask the hard questions about what numbers, if any, there are to support their contention. Admittedly, much of marketing is based on gut feel, hunches, and experience, but there's a lot that is also based on numbers. And don't forget to ask the real question: how “good” are these numbers?”
Bill Schweber , Site Editor, Planet Analog