Advertisement

Blog

Integrated Analog & the Visually Impaired

Commenting on Steve Taranovich’s blog on CES, I said, more or less, that a lot of the connected technology seemed to have been conceived in the scrambled brains of marketing people. I feel somewhat justified in such a pot shot having had the word marketing in my title several times in my career. Nonetheless, thinking about what would be useful applications, it occurred to me that sensors and intelligent processing could really benefit people with disabilities.

In particular, I started thinking about the visually impaired. I considered how they might benefit from advances in the integration of sensing and communications technologies. Here I'm going to expand on a couple of lines of thought, as a starting point to get feedback from the deep knowledge of the Planet Analog community.

Let's start with a basic challenge that might present itself to a visually impaired individual. Most of us have the freedom to go into our kitchens, stand there with the refrigerator open, and see what we want to eat. We also automatically test leftovers and things that have been in there for a while for quality. I'll bet many of you have done the look test (for mold, discoloration, other visual signs for do not eat) and the sniff test (does it smell bad?). Now imagine someone with almost no visual perception — how is he or she to decide if something is OK? Analog integration to the rescue! Figure 1 shows a simple flowchart for a lunch-decision process. Now let's consider how to do this for our hungry, sightless person.

Figure 1

Flowchart for deciding whether or not something is OK to eat for lunch.

Flowchart for deciding whether or not something is OK to eat for lunch.

To accomplish this, we need at least one sensor — a frontend DSP microprocessor — and some way to alert the user of the outcome. Considering we have Google Glass, Wearable Sensors, and the Internet of Everything, I'm proposing a highly integrated solution as the best way to create a product from this. Because of the variety of food and packaging types that might be found in a refrigerator, I want to integrate a hyperspectral sensor to look at the food, plus a secondary optical sensor to read labeling. I'm assuming our person's aroma sensor is working OK, so we won't integrate that.

Hyperspectral imaging goes beyond capturing color images for analysis (which would fall into normal image processing) and collects spectral data at the same time, thus enabling detection of a wide variety of things not visible or poorly discerned by the naked eye. For example, this lecture from the University of Haifa on hyperspectral imaging applications shows you can inspect apples, fish, and citrus fruits for damage and freshness. The spectral wavelengths can extend outside of the normal visual range, from UV at 200 nm well into the infra-red at 2,500 nm. As another, more detailed example researchers at the Catholic University published a paper on hyperspectral imaging of apples to determine quality, which goes into great detail about the methods.

One issue we might have is that the current state-of-the-art in hyperspectral imaging is pretty large systems. Figure 2 shows a bench system from Gooch & Housego, which might be used in research.

Figure 2

Gooch & Housego benchtop spectral-imaging synthesis system(Source: Gooch & Housego)

Gooch & Housego benchtop spectral-imaging synthesis system
(Source: Gooch & Housego)

A bit more portable is Headwall Photonics' micro sensor, shown in Figure 3. This device is targeted at UAVs and similar applications, and weighs less than two pounds.

Figure 3

Headwall Photonics 'Micro' hyperspectral sensor, available in 380 nm to 1,000 nm and 900 nm to 1,700 nm, and under two pounds! (Source: Headwall Photonics)

Headwall Photonics “Micro” hyperspectral sensor, available in 380 nm to 1,000 nm and 900 nm to 1,700 nm, and under two pounds!
(Source: Headwall Photonics)

Of course the underlying sensor is really just a CMOS device. Various approaches can be used to split up the incoming light for hyperspectral analysis. Figure 4 shows one approach offered by Teledyne Dalsa, which filters the light onto several distinct sections on a single chip sensor. Another approach uses a prism to split the light into a spectrum that then illuminates multiple sensors.

Figure 4

Teledyne Dalsa multispectral single chip sensor.(Source: Adapted from Teledyne Dalsa's website)

Teledyne Dalsa multispectral single chip sensor.
(Source: Adapted from Teledyne Dalsa's website)

As daunting as this may seem, this amounts to integrating two CMOS image sensors into a small package. To avoid carrying around a computer and software, let's integrate an LTE modem, and optionally a Bluetooth Low Energy link (if we want to, say, use an LTE phone as the data link, and possibly do some processing there). We'll put all of this into a pair of glasses. Then, we can run all the analysis in the cloud. In block diagram form, this doesn't appear so bad.

Figure 5

First-level integration blocks. We need to power two CMOS sensors, a couple of analog frontends, a couple of special-purpose processors, and the wireless communication and audio section. The audio section could be used to sound tones indicating good or bad, or if enough processing power could be integrated, could tell the wearer something looks spoiled. Using the regular camera for image recognition as well as optical character recognition (OCR) could allow reading bar codes on packaged foods, and looking for the freshness dates.

First-level integration blocks. We need to power two CMOS sensors, a couple of analog frontends, a couple of special-purpose processors, and the wireless communication and audio section. The audio section could be used to sound tones indicating good or bad, or if enough processing power could be integrated, could tell the wearer something looks spoiled. Using the regular camera for image recognition as well as optical character recognition (OCR) could allow reading bar codes on packaged foods, and looking for the freshness dates.

Most of the blocks in Figure 5 are available in highly integrated form. A first approach might be to integrate the hyperspectral sensor, a camera module (like this from ST Micro), a dedicated signal processing chip (to do preliminary analysis from the sensor, reducing communication load), and possibly another processor (such as this one, to do some character recognition locally), and the modem (how about Qualcomm’s Gobi), plus power, into an SiP design.

Of course, I'm oversimplifying. But most of the technology exists, and with some integration, could provide a platform to help hundreds of thousands of visually impaired people in our society (US figures from the National Federation of the Blind). What do you think? How would you approach the design?

Related posts:

41 comments on “Integrated Analog & the Visually Impaired

  1. Scott Elder
    January 17, 2014

    Hi Blaine,

    I've been wishing someone would come up with a simple handheld mechanism to tell me, via sensor (ultrasound?), that the watermelons on the grocery store table are fresh!

  2. Davidled
    January 17, 2014

    I guess that smartphone has capability to the customized App. It could be minimized to fit into the smartphone with small scale sensor similar to health APP. But system should be approved by Food factory.

  3. eafpres
    January 17, 2014

    @Scott–In some ways the hyperspectral imaging work is like electronic signature analysis (by signature I mean some characteristic pattern in frequecy or something else that allows unique determination).  If you could demonstrate ultrasound signatures of fresh melons vs. not fresh, you might really have something.

  4. Scott Elder
    January 17, 2014

    @eafpres,

    Curiously enough, there is a patent on how to scan a watermelon.  The problem is it is based upon something that is practical only for a manufacturer using an expensive piece of equipment.  But I suspect if one poked around a bit, they'd find something with ultrasound that would provide an adequate signature for low cost; something that would fit inside a woman's purse.  I doubt you'd find it hanging in the produce section. 🙂 

  5. Victor Lorenzo
    January 18, 2014

    Just at first sight, without digging too much on this subject, but it appears to me that it could be possible to achieve it for the types of melon I've seen/tasted in several places in Europe.

    The 'sound' of the echo that can be perceived when hitting it with the palm of the hand is different according to its state.

    If it could be implemented on a mobile phone, the operative could be:

    1) place the mobile phone over the fruit,
    2) hit it 'softly' two/three times with the palm of the hand,
    3) see the results on the screen and decide to buy that one or go for another one.

    …supposing the phone's mic has enough bandwith for capturing the echo signals.

    The camera could be used for detecting the fruit type and load the appropiate profile.

     

  6. RedDerek
    January 22, 2014

    Good thought process, but rapping the fruit can be done at different levels. How about having a second phone tied to the first via BlueTooth. Place a phone on either side of the melon and press the start button. The transmitting phone sets the proper amplitude and frequency of the “rap”. The second measures the timing and echo data.

    Granted, the system would need some dimensions to ensure it is reading the results properly.

  7. goafrit2
    January 22, 2014

    @Scott, excellent idea but if the market is really big, these guys in Silicon Valley can do so with mobile apps and smartphone cameras. It may not be too hard if there is a strong market that people eat watermelon and need a tech-driven way to ascertain freshness.

  8. goafrit2
    January 22, 2014

    You do not need a customized sensor for this. A really very good camera like the 48MP you see in Nokia Lumia will do the job. If mobile apps can record the values of resistors via color and help detect diseases via the texture of the skin, I think watermelon will not be extremely challenging.

  9. goafrit2
    January 22, 2014

    >>  If you could demonstrate ultrasound signatures of fresh melons vs. not fresh, you might really have something.

    That should not be very challenging to do. Farmers are using sensors to know when their crops are ready for harvest. That means they have a database that can discriminate between ripe fruits and unripe ones. One can so same for any fruit with small budgets. Go to the market, collect pictures of watermelon. Ask people to validate by tasting if those melons are fresh or not. Over time, you can build a database with numbers to indicate degree of freshness or not.

  10. eafpres
    January 23, 2014

    @goafrit2–I would say that for most fruit you cannot tell with much accuracy the ripeness with just looking (or a picture taken in normal spectrum). That is the promise of hyperspectral imaging–you can “see” things that are not visible to the eye, and various studies have shown they can detect food spoilage much earlier.

    The acoustic testing is interesting, though.  I speculate that the velocity of sound propagation through a melon changes when it ripens.  Also the dispersion might be different.  If the input could be thumping the melon and the smartphone could hear both the sound of the thump transmitted through air and the sound transmitted through the melon, then you could do time of flight (TOF) measurements etc.  As noted you might need to enter the size of the melon for such a measurement.

  11. jkvasan
    January 24, 2014

    Blaine,

    Great post with practical possbilities.

    Often, the visually or physically challenged are the target users for many electro-medical devices and any innovation is focussed towards those. Hearing impaired are considered to be physically challenged but except for the hearing aids and some testing devices nothing much is available for them as gadgets. I would like to propose a device which can alert the person while crossing the road by processing the horn sound of the approaching vehicle. This alert can be something like a smartphone vibration.

  12. eafpres
    January 24, 2014

    @Jayaraman Kiruthi Vasan–thanks for your thoughts.  Using a smartphone to augment hearing seems very practical.  

  13. jkvasan
    January 24, 2014

    eafpres,

    Sometimes, hearing and speach impairments come as a bundle. Just because they cannot hear they do not speak. Often, addressing hearing problems could automatically solve the speach problem. 

    Another interesting application would be pain killing through electricity. If we have a single chip solution for this, it would be a boon to sports persons and the like. I wrote a blog on this subject in Microcontroller Central which is now defunct.

  14. amrutah
    January 26, 2014

    Jayaraman,

       This is a very practical solution of using a smartphone to alert a hearing impaired person.  A smart watch can also be used where in you can make it glow with different colours.  Also for low-cost purposes a small arm band could also be devised.

  15. SunitaT
    January 27, 2014

    I wrote a blog on this subject in Microcontroller Central which is now defunct.

    @JAYARAMAN, interesting application. Is it possible to repost that on this site ? I am curious to know if devices based on this idea are already available ?

  16. jkvasan
    January 27, 2014

    Amrita, I was always wondering what would one do with a smart geae/watch. I agree, this could be a potential application.

  17. jkvasan
    January 27, 2014

    Sunita, I don't know if that is possible because it is upto the editors, Rich and Bras to decide on publishing it again. Also, it's core focus was on mcus and not on Analog electronics.

  18. jkvasan
    January 27, 2014

    Sunita, I don't know if that is possible because it is upto the editors, Rich and Brad to decide on publishing it again. Also, it's core focus was on mcus and not on Analog electronics.

  19. SunitaT
    January 27, 2014

     I don't know if that is possible because it is upto the editors, Rich and Brad to decide on publishing it again.

    @JAYARAMAN, Thanks for the reply. I wish all the blogs of Microcontroller Central are restored because there was so much of data available on that site.

  20. SunitaT
    January 27, 2014

    A smart watch can also be used where in you can make it glow with different colours.

    @amrutah, I think we can also add vibrator to this app so that if the user fails to see the screen he can always sense the vibrations.

  21. SunitaT
    January 27, 2014

    The 'sound' of the echo that can be perceived when hitting it with the palm of the hand is different according to its state.

    @Victor, I think similar thing is used to check the quality of coconut. You can find the amount of water inside the coconut by hitting it with the palm. I think the app you mentioned definitely helps us to identify fruit type.

  22. SunitaT
    January 27, 2014

    Go to the market, collect pictures of watermelon. Ask people to validate by tasting if those melons are fresh or not.

    @goafrit2, I think using pictures alone to measure the freshness will be very difficult. I think we need different sensors to measure the freshness of the frutis.

  23. jkvasan
    January 27, 2014

    Sunita, Microcontroller Central and The Connecting Edge were great sites and buzzed with fervent activity. The blogs and comments were of high quality.

  24. jkvasan
    January 27, 2014

    Sunita, Many of those smart watches have vibration alerts already, I guess. Days are not far when some James Bond style watches would be available across the counter.

  25. Victor Lorenzo
    January 27, 2014

    @SunitaT, in the case of coconut my mother showed me a method based on the sound and inertia feeling produced by the water when shaking it. In that case you don't need it to have the exocarp removed.

    Well, it's just for curiosity as it can not be implemented in highly integrated real life applications.

  26. amrutah
    January 27, 2014

    ” I think we need different sensors to measure”

    The water content of a fruit is another measure of the freshness.  Now since we have probes that can detect the presence of water on moon/mars, checking the water content of the fruit is possible, but the only problem is the affordability.

  27. SunitaT
    January 27, 2014

    Microcontroller Central and The Connecting Edge were great sites and buzzed with fervent activity.

    @JAYARAMAN,  I totally agree with you. Blogs were very informative, I really hope those sites are relaunched again so that it helps all the student/professional community.

  28. SunitaT
    January 27, 2014

    Now since we have probes that can detect the presence of water on moon/mars, checking the water content of the fruit is possible, but the only problem is the affordability.

    @amrutah, I totally agree with you. I think finding the water content definitely helps us to find the freshness of the fruit. Affordability is definitely a concern but I am sure we will invent new sensors which will affordable and reliable.

  29. Victor Lorenzo
    January 28, 2014

    Simply another thought, perhaps not relevant at all.

    Ethylene emissions occur from vegetal cells as a normal process, it is not soluble in water and acts as a phytohormone with direct effects on fruit rippening. It seems to me like a possible gas sensor application.

    At the end it could be a device with a high resolution camera, an ultrasounds emitter-sensor pair and a gas sensor (probably an array of them). It could be fun to have such a platform for experimentation.

  30. goafrit2
    January 30, 2014

    >> . I think we need different sensors to measure the freshness of the frutis

    Why the need of sensors. How are you going to use them? Would that be invasive or not? I think a good atlas of these fruits can build enough analytical model that will be useful to make the measurement a science. Of course a camera is a sensor, just that we have a name for it. 

  31. PCR
    January 30, 2014

    Amrutah I believe that the best option for the hearing impaired is the vibrate option, 

  32. PCR
    January 30, 2014

    SunitaT I also do agree with you on this, They maintain the quality of the site. I used to refer those sites for my references even thoue I am not a member. 

  33. PCR
    January 30, 2014

    Yes SunitaT finding the water content will be a option and also finding a air content in the fruit also will be helpful. And the affordability is also count more. 

  34. amrutah
    January 30, 2014

    @Victor: “Ethylene emissions occur from vegetal cells as a normal process”

       This is a nice idea for detecting the ripening of a fruit or on the way of getting rotten.  But not all fruits emit ethylene and the device might get limited to a few fruits and vegetables.

       To add, we can have something like Electronic Nose.  The database stores the smell data and the we just swipe over the fruit to smell it using the e-nose and trigger if its ripe.

  35. SunitaT
    January 31, 2014

    But not all fruits emit ethylene and the device might get limited to a few fruits and vegetables.

    @amrutah, I thought all the fruits emit ethylene. Can you name some fruits which dont emit ethylene ?

  36. PCR
    January 31, 2014

    But Amruta Electronic Nose I still under the research stage, Isn't it?

  37. yalanand
    January 31, 2014

    But Amruta Electronic Nose I still under the research stage, Isn't it?

    @Ranasinghe, Electronic nose instruments are already used by research and development laboratories, quality control laboratories and process & production departments for various purposes.

  38. amrutah
    January 31, 2014

    “Can you name some fruits which dont emit ethylene ?”

    @SunitaT: Not all fruits emit ethylene nor do they get ripened if exposed to ethylene.  Basically most of the yellow coloured fruits emit ethylene and also gets ripened.

    When I was a kid I used to wonder why do we store bananas, mangoes, papaya rolled in gunny bags or in wraps with hay or straws for ripening.  When wrapped with these, the ethylene concentration increases and bootstraps the ripening process.

      This is one site that lists fruits and vegetables that get ripened or rotten by ehylene.

  39. PCR
    February 3, 2014

    Yalanand, It a new thing for me what are the other purpose of having an electronic Nose? Can you please share some links of a research text. 

  40. Victor Lorenzo
    February 3, 2014

    @amrutah, Thanks for the link.

    When I was a kid I used to wonder why do we store bananas, mangoes, papaya rolled in gunny bags or in wraps with hay or straws for ripening.

    You can try this at home, put some seeds of “calabaza” including part of the fibers surrounding them inside the bag with the bananas. The rippening process will be noticeably speeded up.

    I suggested sensing ethylene based on some of its characteristics, it is a very active fito-hormone, strongly related to fruit rippening processes, has low solubility in water and is produced in major quantities when the fuit seeds are mature.

     

  41. jack90
    March 8, 2014

    Integration of latest technology such as berth credit, impediment detection, moreover fixed computers complys the draft of a navigation computer to abet the visually faulty sail readily.

    annotated bibliography online

Leave a Reply