I've always regarded debugging, especially of circuitry hardware, as the most difficult of engineering challenges. It's not taught in school (for many reasons), there are very few books about it, and it takes bench and field experience (preferably with a seasoned mentor) to get good at it. Just about every engineer with any experience has some fascinating anecdotes (a.k.a. horror stories) about various debugging challenges he or she faced.
There is one excellent book on the subject, written by an engineer for engineers: Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David J Agans. What's really odd is that it is published by the American Management Association, an organization which is pretty far removed from down-and-dirty engineering design and debug. Nevertheless, it is a great book.
Back in the day, when systems were mostly hardware circuitry with perhaps a little software to guide them along, you could at least make good headway in debugging by following the signal flow from input to output. Those days are gone, and they are not coming back. Instead, now, there is constant interaction between the hardware and software — a back-and-forth dialogue — which means straightforward linear signal flow and causality are often non-existent.
Is it the hardware when a circuit sub-function appears not to be working properly? Or is the software not setting up and responding to the hardware? Or maybe it's both? Maybe there really isn't a problem, in the strict sense of the word, but instead you have a misconception or misunderstanding of what the circuit and system are supposed to be doing?
Rules for debugging are hard to come by, and most have many exceptions, but there are some starter rules I try to follow:
- Stop, think, and don't rush, difficult as it may be to slow down.
- Take notes of what you think you see and what steps you are taking to investigate — often hard to do in the heat and pressure of the debug process (often a crisis mode).
- Check the power supply rails in many places, to make sure they meet nominal specifications and are clean.
- Have someone else come by and see if the individual observes the same symptoms that you see. Also have the person re-check the supplies. (I once spent a morning debugging an analog circuit which had a very random output, only to eventually realize I neglected to turn on the power — ouch! )
Although debugging has always been difficult, I think it is now even more so as ICs become more highly integrated. Why? Prior to higher levels of integration, you could at least separate functionally and, often literally, just about all analog and I/O circuitry from the processor and its software. That allowed you to test out the analog part independently. The same was true for the processor and software side.
To facilitate this separability, I tended to choose analog and peripheral ICs which would initialize themselves in a standard, well-defined state (such as gain of +1 for a software-controlled programmable gain amplifier) in the absence of any set-up or configuration by the processor. That way I could test the circuit without processor and software. Similarly, I would look for digital ICs which powered-up in a known state so that I could verify their operation as standalone components.
But as more of the analog, digital, and processor/software functions have been packed into a single IC, it's no longer possible functionally or physically to disconnect these blocks. Still, you may be able to run them independently via some pin-selection options on the IC, or through other techniques, depending on the ICs you choose.
Regardless, debugging systems which are centered on highly integrated components is a real challenge, and one that calls for new strategies and tactics, along with appropriate test equipment and analyzers, test suites, and more. Equally important, you have to establish a smarter debug plan at the design phase, so you are not looking at the prototype PC board and saying, “We should have planned for XYZ,” or, “We really should have added these 'hooks' so we could see what's happening.”
How have you changed your prototype design process when you are using highly integrated ICs, to accommodate the inevitable debug cycle? How have you adapted your debug tactics to this present-day reality?
- Reference Designs: Strong Start but Weak Finish?
- Can We Have Too Many Op-Amp Performance Curves?
- Functional Integration Changes Your Calibration Strategy, Part 1
- Functional Integration Changes Your Calibration Strategy, Part 2
- Simulating Mixed-Signal ICs: Good or Bad News?
- Point/Counterpoint: Let’s Debate Integration
- When an Improvised Solution Is ‘Good Enough’
- Did You Say ‘Programmable Analog’?
- What Analog’s ‘Imperfections’ Taught Me
- Getting Over My New Integration Fears