Advertisement

Analog Angle Blog

Debugging Tactics: 4 Ways to Increase Integration

I've always regarded debugging, especially of circuitry hardware, as the most difficult of engineering challenges. It's not taught in school (for many reasons), there are very few books about it, and it takes bench and field experience (preferably with a seasoned mentor) to get good at it. Just about every engineer with any experience has some fascinating anecdotes (a.k.a. horror stories) about various debugging challenges he or she faced.

There is one excellent book on the subject, written by an engineer for engineers: Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David J Agans. What's really odd is that it is published by the American Management Association, an organization which is pretty far removed from down-and-dirty engineering design and debug. Nevertheless, it is a great book.

Back in the day, when systems were mostly hardware circuitry with perhaps a little software to guide them along, you could at least make good headway in debugging by following the signal flow from input to output. Those days are gone, and they are not coming back. Instead, now, there is constant interaction between the hardware and software — a back-and-forth dialogue — which means straightforward linear signal flow and causality are often non-existent.

Is it the hardware when a circuit sub-function appears not to be working properly? Or is the software not setting up and responding to the hardware? Or maybe it's both? Maybe there really isn't a problem, in the strict sense of the word, but instead you have a misconception or misunderstanding of what the circuit and system are supposed to be doing?

Rules for debugging are hard to come by, and most have many exceptions, but there are some starter rules I try to follow:

  • Stop, think, and don't rush, difficult as it may be to slow down.
  • Take notes of what you think you see and what steps you are taking to investigate — often hard to do in the heat and pressure of the debug process (often a crisis mode).
  • Check the power supply rails in many places, to make sure they meet nominal specifications and are clean.
  • Have someone else come by and see if the individual observes the same symptoms that you see. Also have the person re-check the supplies. (I once spent a morning debugging an analog circuit which had a very random output, only to eventually realize I neglected to turn on the power — ouch! )

Although debugging has always been difficult, I think it is now even more so as ICs become more highly integrated. Why? Prior to higher levels of integration, you could at least separate functionally and, often literally, just about all analog and I/O circuitry from the processor and its software. That allowed you to test out the analog part independently. The same was true for the processor and software side.

To facilitate this separability, I tended to choose analog and peripheral ICs which would initialize themselves in a standard, well-defined state (such as gain of +1 for a software-controlled programmable gain amplifier) in the absence of any set-up or configuration by the processor. That way I could test the circuit without processor and software. Similarly, I would look for digital ICs which powered-up in a known state so that I could verify their operation as standalone components.

But as more of the analog, digital, and processor/software functions have been packed into a single IC, it's no longer possible functionally or physically to disconnect these blocks. Still, you may be able to run them independently via some pin-selection options on the IC, or through other techniques, depending on the ICs you choose.

Regardless, debugging systems which are centered on highly integrated components is a real challenge, and one that calls for new strategies and tactics, along with appropriate test equipment and analyzers, test suites, and more. Equally important, you have to establish a smarter debug plan at the design phase, so you are not looking at the prototype PC board and saying, “We should have planned for XYZ,” or, “We really should have added these 'hooks' so we could see what's happening.”

How have you changed your prototype design process when you are using highly integrated ICs, to accommodate the inevitable debug cycle? How have you adapted your debug tactics to this present-day reality?

Related posts:

9 comments on “Debugging Tactics: 4 Ways to Increase Integration

  1. Davidled
    November 14, 2013

    When I was junior engineer, my supervisor told all engineer to clean up Lab desk before testing and debugging any engineering work. They told us that in most cases, some electronic debris including piles of wires which is not used, should be on the other shelf. In the end, all engineers test and debug any PCB in the very clean table.

  2. etnapowers
    November 15, 2013

    “Although debugging has always been difficult, I think it is now even more so as ICs become more highly integrated”

    I agree on that. One of the fundamentals of the debugging activity is to make one step at a time, to be sure to associate a specific problem to a specific block, which should be the only one under test at that moment. Today this activity of insulating a single block per time is very difficult because of an high grade of integration.

     

  3. etnapowers
    November 19, 2013

    @DaeJ: The test and debug phase is a key step in the engineering process. Having a clean table to do that avoids many problem like ESD discharges, soldering issues… I agree with the choice of your supervisor that time.

  4. etnapowers
    November 19, 2013

    “Stop, think, and don't rush, difficult as it may be to slow down” : This strategy is a good thing in general. It allows to consider all the aspects of the problem and avoid to follow a wrong way, based on a first supposition that may be a wrong assumption, due to an idea based on  instinct.

  5. etnapowers
    November 19, 2013

    “Take notes of what you think you see and what steps you are taking to investigate — often hard to do in the heat and pressure of the debug process (often a crisis mode).”

     

    To save time , sometimes it's necessary to spend some time in taking notes of the data at your disposal and of the steps taken to check a problem. Only when correlating the data and the steps in a way that makes sense , it's a good sign that the solution is close to be found.

  6. etnapowers
    November 19, 2013

    “Check the power supply rails in many places, to make sure they meet nominal specifications and are clean.”

     

    It's another good rule when debugging, the checking of the signal path, by measuring current voltage in many test points (previously designed on the test board). The instruments utilized for debug have to be calibrated too as the power supplies.

  7. etnapowers
    November 19, 2013

    “Have someone else come by and see if the individual observes the same symptoms that you see. Also have the person re-check the supplies.”

     

    It's a golden rule, a problem can be interpreted by an engineer in one mode, but only if another engineer (or many other) sees the same problem in the same mode, the engineer can be sure that his interpretation is objective. Re-checking the power supplies is important too, normally there is a calibration personnel deputed to check and calibrate all power supplies.

  8. RedDerek
    November 19, 2013

    Another method is to pull some parts in the stream to try to isolate what is not working properly. At my office, I had a tech come in with a perplexing supply loading issue – something was pulling the rail low. I suggested to pull some components in the attempt to isolate the loading. This allowed the tech to hone in within the next 5 minutes to a TSOP6 MOSFET inserted backwards.

  9. yalanand
    November 30, 2013

    In present days not only debugging but most of the work is also done by tools itself. The design also mostly done by tools. Because of this there is lack of debugging capabilities in young engineers because they dont get chance to debug the circuits.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.