Advertisement

Blog

Unintended Consequences

Editor’s note : To complement Bill Schweber’s blog Beware the unintended consequences of more-efficient design, author Aubrey Kagan has submitted his blog which was previously posted on EETimes Microcontroller Central.

In South Africa the fire protection sprinkler system has a different model to the one used in North America. When the sprinkler system is activated, a pump must start up to keep the water network pressurized. When the Fire Department arrives at a fire the first thing they do is to shut off the mains to the building, so there must always be a diesel engine. This is dictated by the regulations set up by some umbrella organization established by the insurance industry. An electric motor is permissible, but not necessary.

The philosophy of protection consisted of attempting to start the diesel engine, and keep it running at all costs when a was detected. The diesel engine had two batteries and the start process was to crank using one battery for a period. If the engine did not start there would be a short delay and then an attempt would be made to crank using the other battery with the same delays. If the engine was not running the crank sequence would be repeated so there were 3 attempts on each battery. Six failed crank attempts would generate an alarm. If the engine started up, but then failed the whole process would be repeated. With an electric motor driving a second pump, keeping the system pressurized was a lot more elegant, rather than having the diesel start up all the times as the pressured decayed.

The whole system was activated by a drop in pressure of one or more water pressure sensors in the water line. The sprinkler heads in South Africa work differently to those demonstrated on Hollywood cinema screens. Each head consists of a valve held shut by a glass bulb. In the glass bulb is a liquid that expands significantly when heated and that breaks the bulb allowing the water to start flowing. This means that for a small heat source, only one sprinkler may be activated.

When we started designing this product, the industry worked around relay logic for the controllers. We decided that a micro would be the better way to go, provided we managed to pass the safety tests. This was the start of my interest in watchdog timers and other techniques for robust operation of a microcontroller based system. I designed a single board built around the 8748 family which included three lead-acid battery chargers (the third for the system standby battery), a fairly large LED driver matrix for about 20 LEDs relays and drivers to crank the motor and drive klaxons and input circuitry for engine RPM and water pressure. Later we changed to an 8051 with an on board LED matrix display to reduce the wiring.

The original design in its panel. There was quite a bit of wiring associated with the front panel LEDs.

The original design in its panel. There was quite a bit of wiring associated with the front panel LEDs.

The panel connected to the diesel engine. The visible silver shaft on the engine would be coupled to the water pump on site. Despite the fact that this particular unit was installed in an oil refinery, note the attached fuel tank.

The panel connected to the diesel engine. The visible silver shaft on the engine would be coupled to the water pump on site. Despite the fact that this particular unit was installed in an oil refinery, note the attached fuel tank.

The revised design with the LEDs on the PCB. In this approach the whole controller was mounted on the door of the panel. Note the dual LEDs for redundancy.

The revised design with the LEDs on the PCB. In this approach the whole controller was mounted on the door of the panel. Note the dual LEDs for redundancy.

The product proved remarkably successful since it was being included as part of a package offered by the pump and piping manufacturer. In one installation the water supply was drawn from a Braithwaite tank (a water tank built from panels and then waterproofed with bitumen). It used a large Caterpillar engine with some instrumentation (oil pressure, water temperature etc.) mounted directly on the engine block. There was no additional electric motor driven pump.

As far as we know the sprinkler system had a small leak, but in any event the pressure dropped and the diesel sprang into action. Unfortunately some bitumen was sucked into the water outlet from the tank and the water supply was blocked. The design was such that the water was circulated through the engine (as cooling) before being pumped into the sprinkler pipes. I am sure you feel the tingle of impending doom! With no water pressure the diesel started up and then after a short while would overheat and shut down (of its own accord). Once stopped, our system cranked it again and it started up again getting ever hotter until the engine seized completely. The gauges on the block literally melted resembling Salvador Dali’s painting “The Persistence of Memory” (The Persistence of Memory by Salvadore Dali ). All of this of course was forensically determined, but while it was being assessed we were terrified as to where the responsibility lay.

Have you ever been visited by the Law of Unintended Consequences?

5 comments on “Unintended Consequences

  1. Rama Murthy
    February 3, 2016

    I am sure the system would have had some sort of water sensor which would have triggered the Klaxons for no water alert. But the amount of effort to realise a fool proof system is quite high.

  2. cookiejar
    February 4, 2016

    The more things that can go wrong, the more things will go wrong – a Murphy law?

    Remember the good old days when every large industry had a water tank, high in the sky proudly heralding the company logo?

    That was to supply the fire sprinkler system, with gravity providing the pressure and feeding any sprinkler head that opened.  Large diameter pipes were the rule.  Simple, elegant and about as reliable as you can get, but boring to design.

    A fire sprinkler system spends most of its lifetime in standby and the expectation that it will work when called for is very high.  A complex diesel powered system can be a reliability nightmare, requiring rigorous routine maintenance and testing.  Diesel or gas fuel doesn't age well.  These days propane or natural gas is the fuel of choice.

    I recall the tale of the main NORAD deep underground bunker, responsible for the security of North American during the Cold War.  For their AC backup they had 3 large diesel generators.  Keeping up with the Jones', the military switched to the high cost NiCad batteries as they required less maintenance and were promoted as far more reliable that lead acid batteries used previously.  Once one generator started it could be used to help crank the others.  Then one day there was an actual power failure.   The NiCads surprisingly suffered their now famous “memory effect” having been on trickle charge for months and none of the generators could start.

    The solution was to get away from unreliable electrics and electronics.  They installed air compressors and air starters on each engine.  The starting energy was stored in pressurized air tanks.  Cheap, mechanical and proven far more reliable that new-fangled electrics and electronics.

    Then there's the rapid obsolescence of active electronic devices.  Flash memories that program everything have only 10 year specified retention.  You can guarantee that nothing electronic designed today will ever be a functioning antique.   Even if you had a fresh flash, how would you get your hands on the listing?  Model T's will outlive our designs by centuries.  All my dozen Heathkits from the 1950s are still fully functioning today.

    Most reputable camera manufacturers now only support a new model for 2 years.  After that they destroy their parts inventory. 

    How does it feel to be designing throw away junk?

  3. michaelmaloney
    October 31, 2018

    It is interesting to know how different nations practice different safety measures according to their local needs. I guess should a firefighter from another location were to apply to work elsewhere, he/she should be sent for training even after having served for several years. Different rulings call for different understanding and that is important to avoid wrong procedures during an emergency situation.

  4. ChristopherJames
    February 22, 2019

    Gosh I'm looking at those circuit boards and wiring and I'm amazed at home someone managed to put them all together for the safety of people. I'm just saying that if it were me, I think I'd be manually carrying buckets from the nearest creek of well or something! Haha! Thank goodness for the great engineering minds who've come up with such a device to help everybody right?

  5. AubreyKagan
    February 23, 2019

    Christopher

    Wiring is something that often gets overlooked. I wrote a blog on “Tidy Your Wiring“, but I think if you follow the links posted in both my comments you will see quite how messy things can become.

     

     

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.