I've always felt that system-level debugging is among the most difficult of all engineering challenges. It requires experience and mentoring (neither of which you'll get in school); the ability to think both outside and inside the box; the stamina to examine and re-examine all your assumptions, data, and tests; and use of multiple disciplines as you work through a problem that may involve one or more of circuitry, software, and mechanical functions.
And then there those dumb but simple problems that turn out to not be problems at all, just a misreading of the observed symptoms.
I recently had some intermittent buttons on the keypad of a TV remote control—one of those $10 universal ones you can buy anywhere. Sure, I could have just tossed it out, but that's no fun. I had opened this one before, and the problem was one I've seen on many such remotes, cordless phones, and similar units.
They use a keypad assembly made of conductive elastomer dots which are supposed to complete a circuit on the PC board. In time, the dots start oozing some gunk as the dot material begins to decompose, and the gunk inhibits any low-resistance circuit closure through the dot.
I have a two-phase repair process: first, I clean the dots and the associated PC board with alcohol or degreaser. If that doesn't work, I glue small circles of aluminum foil onto each elastomeric contact, so real metal is now closing the circuit, while the gunk can no longer interfere. Sure, it's a lot of work, but it is oh-so-satisfying.
But this last go-round had me puzzled for a few minutes. Here's why: I had the remote control mounted in an angled vise—the kind you use when working on PC boards—so it wouldn't move around as I worked on it. I went through my usual cleaning steps, and was ready for a check of all the keys on the unit.
This remote control has a tiny LED in the upper corner, which flashes whenever you push any button, which gives a nice bit of user feedback, and is also handy for checking that I cleaned all the keys. Wait a moment: the LED was on, even when no keys were depressed. I figured that one or more of the keys was stuck “on”, so I checked the PC board for shorts, a stray filament of wire, anything. Everything seemed OK.
So, I next took out the battery (in preparation for some continuity checks with the ohmmeter), and the LED stayed on even then. This meant one of several things: perhaps the unit included a supercap along with the battery, or it had some sort of harvesting circuit—but those were unlikely in such a cheap, toss-away unit. Or I had found my path to fame and fortune, with an LED which was self-powered and needed no external source (infinite efficiency!), but that too was unlikely. Hmmm…maybe I was misinterpreting the situation?
Then reality came, as I inadvertently moved my hand between my bench lamp and the remote control—and the LED went out. Long story short: that little LED was capturing and reflecting the illumination from the bench lamp, so it looked as if it was a light source, when it was not. Since the position and angle between the lamp and the LED was not changing, the LED was reflecting and thus “on” all the time.
It was a simple optical effect, and the cleaned-up keys of the keypad were actually fine. The reason I didn’t see the LED come on as each button was pushed was that the LED's internally reflected glow from the bench lamp was much stronger than the powered-on illumination of the LED; in effect, the LED's powered output was swamped by the reflection it was delivering (it's a variation of the theme of noise>signal).
This is clearly a fairly trivial debug problem . While I assumed at first that it was a problem with the electronic circuitry, or mechanical buttons on the keypad, it was neither. Instead, it's another lesson for my learning-experience logbook, and fortunately it's a low-cost one.
Have you ever had debug problems where the evidence and your assumptions proved to be way off from the reality? And how did you find out? ?
[BTW, if you are interested in learning how to be better at debugging, check out this book excerpt we published a few years back, “Debugging: The 9 Indispensible Rules for Finding Even the Most Elusive Software and Hardware Problems.” Even better, get the book and read it—it's well worth your time and the price is very modest. ]