Advertisement

Blog

Can Your Device Recognize Gestures?

In my past life with Intersil, I had occasion to show customers an eval board that was set up to do gesture recognition. It used some of Intersil proximity sensors. It consisted of an array of sensors arranged to detect whether a hand was waved left-to-right or up-and-down. With just that two-axes information, you could arrange to control a variety of devices. Quite useful in automotive applications, though I'm not sure it was ever pursued beyond the eval board stage.

(Source: UC Berkeley)

(Source: UC Berkeley)

Recently, I saw a video from the UC Berkeley school of engineering that showed the work the folks there were doing on gesture recognition using ultrasonics. They claim that their technology is far lower power than the typical optical measuring methods. They compared their version to a camera and claimed 400μW vs. 1W power draw. The prox-sensor method that I cited above doesn't need 1W, but it's still rather power hungry, so this ultrasonic version is of interest for portable, battery-operated devices.

From promotional material from the UC Berkeley, there is this background info:

Researchers at the University of California, Berkeley have developed an ultrasonic graphic recognition system that can track a user's movements and translate them into inputs to an electronic device. Their ultrasonic 3D range sensor system uses batch-fabricated micromachined aluminum nitride ultrasonic transducer arrays and custom CMOS electronics. This technology may be useful in the development of practical gesture-controlled computer interfaces. Optical 3D imagers for gesture recognition, such as Microsoft Kinect, suffer from large size and high power consumption. Their performance depends on ambient illumination and they generally cannot operate in sunlight. Ultrasonic gesture recognition systems measure sound waves to turn gestures into inputs, allowing for comparable performance to optical systems but with less power consumption and environmental restrictions.

This method of sensing should lend itself to large-scale integration. If we could combine the ultrasonic sensor array, the transmitter driver, the receiver pre-amps, and the processing circuitry onto a chip, we'd have a very useful device. Conceivably, it may not be practical to put the entire ultrasonic sensor array on an IC. They may require too much area and may need to be spaced too far apart. The alternative is to: a) keep the transducers completely separate from the circuitry described above, but with all the circuitry still integrated; or b) put just the transmit driver, the receiver preamp, and one ultrasonic transducer on an IC. This is still a useful amount of integration to squeeze on one piece of silicon.

Have you worked on any similar gesture recognition devices? What problems did you have to deal with?

— Brad Albing, Editor-in-Chief, Planet Analog and Integration Nation Circle me on Google+

Related posts:

25 comments on “Can Your Device Recognize Gestures?

  1. eafpres
    December 30, 2013

    Hi Brad–In the past I've worked in corporate office environments with large shared printers and copiers.  It was common knowledge that these systems could not only recognize gestures but interpret facial expressions and act based on the emotional state of the user.  For instance, displaying frustration while waiting for a printout would inevitibly result in it getting “stuck in the queue”, while displaying any level of fear or anxiety would cause varying levels of jams.  It appears these complex systems developed these recognition features spontaneously as they never appeared in any of the marketing literature for the machines.  I'm not sure if further integration would make these systems more or less civilized.

     

  2. eafpres
    December 31, 2013

    One comment on the Kinect system–as I understand it, the system uses active IR illumination with specific patterns which are interpreted upon reflection.  As such, it isn't directly dependent upon ambient illumination, although it likely does suffer degration in sunlight due to high levels of incident IR bombarding the sensor.

    I'll have to go read up on the ultrasound technique.  My guess would be it is very limited in range from the sensor array, at reasonable power levels.  The other question about integration would be are there mechanical issues.  If you think about ultrasonic cleaners, they have relatively large transducers coupled to the cleaning tank in order to have maximum transmission into the cleaning medium.  In my experience those transducers suffer failures, and generate audible sound, vibrations, and heat.  

  3. Netcrawl
    December 31, 2013

    @Brad great post! thanks for sharing it,@easpres thanks for a great commment, most of today's gesture recognition technologies follows image-based approaches, which requires advanced image processing platforms mostly used cameras for capturing gesture, and emplyong these kind of technologies canbe very expensive and there's also some technical issues such as power consumption and data collection.

  4. Brad Albing
    December 31, 2013

    @eafpres – I believe i worked with one of those printers. I recall printing out my updated resume just as my boss walked up. I must have looked anxious because I think that was when the machine jammed. So yes, gesture recognition of a sort.

  5. Brad Albing
    December 31, 2013

    @eafpres – quite right on the ultrasonic cleaners re size and heat. For these gesture recognition devices tho', the size is tiny — per that photo and per this video:

    Ultrasonic Gesture Recognizer for Portable Electronic Devices

    As you noted, range would be limited, so not like those Kinect systems. But just right for the wave of a hand at perhaps 5 to 20cm.

  6. eafpres
    December 31, 2013

    Thanks, Brad.  I do wonder at the reliabiilty of those micro transducers.  I think we are heading full speed into adoption of nano-everything; time will tell what methods are solid and what aren't.

    By the way, kudos to the UC Berkely PR department on the nice video!

  7. SunitaT
    December 31, 2013

     If we can combine the ultrasonic sensor array, the transmitter driver, the receiver preamps, and the processing circuitry onto a chip, we'd have a very useful device.

    @Brad, thanks for the post. If we are able to integrate sensors and drivers on a single chip then the cost the system would comedown drastically. Moreover single chip solution makes it easy for the end users to incorporate this technology in wide variety of applications.

  8. SunitaT
    December 31, 2013

    For instance, displaying frustration while waiting for a printout would inevitibly result in it getting “stuck in the queue”, while displaying any level of fear or anxiety would cause varying levels of jams.

    @eafpres, thanks for sharing your experience. I never knew printout would respond to expression of the end users. But what is the advantage of making it more inefficient based on the end user expression ?

  9. SunitaT
    December 31, 2013

     As such, it isn't directly dependent upon ambient illumination, although it likely does suffer degration in sunlight due to high levels of incident IR bombarding the sensor.

    @eafpres, I havent heard any complaints which suggests that Kinect system suffers degradation under sunlight. I think it also uses camera setup along with IR which makes it more reliable.

  10. SunitaT
    December 31, 2013

    which requires advanced image processing platforms mostly used cameras for capturing gesture, and emplyong these kind of technologies canbe very expensive

    @Netcrawl, True power consumption is one of the major issues. If we can create low power gesture-recognition systems then we can embed it wide variety of applications.

  11. Vishal Prajapati
    January 2, 2014

    This is a one of the good inventions that can change the game. I think if we can squeeze it down in a single chip, it will be very useful sensor for the mobile and other portable devices. If I am not wrong, right now samsung and other phones are using IR signal to detect proximity of hand or ear. That sensor can be also used for the hand gesture recongnition.

  12. Brad Albing
    January 2, 2014

    @eafpres – yep – that did look pretty professional. But I would expect that coming from that school, they would know what they're doing regarding promotion.

  13. eafpres
    January 2, 2014

    @SunitaT–Generally Kinect is used in-doors and not in direct sunlight.  Such light also makes the screen hard to see, and thus getting a good looking screen goes hand in hand with relatively low ambient light levels.  However, I suspect it would suffer on really sunny days out in a park, if, of course, you could get power to it.

  14. eafpres
    January 2, 2014

    @SunitaT–“But what is the advantage of making it more inefficient based on the end user expression”

    Inside of a large copier there is a semi-intelligent processor that by design likes attention.  By failing at critical moments, much like a child who decides to scream once you take a phone call, the machine gets extra attention.  In fact, the more urgent need of the user, and the more severe the jam, the more help the user calls in.  Soon there were typically a half-dozen or more well-paid professionals looking into the innards of the things, moving levers back and forth, etc.  How satisfying that must be for the narcissitic processor in those copiers.

  15. Victor Lorenzo
    January 3, 2014

    @Vishal: “(…)it will be very useful sensor for the mobile and other portable devices

    For my daily use I don't see how this kind of sensor will improve my mobile phone usage experience, but I do find it usefull as a HMI device for computers and TV sets for browsing, reading documents, chaning channels, modifying speakers volume, and alike.

  16. Victor Lorenzo
    January 3, 2014

    @eafpres: It is an interest point when you say that “Inside of a large copier there is a semi-intelligent processor that by design likes attention “, I agree with you and if you let me I would say that also applies to modern cars.

    Those things are rather expensive so at the end it is very disappointing the final end-user experience resulting from their processors “intelligence” (or lack of).

  17. Davidled
    January 3, 2014

    I wonder whether semi-intelligent processor is in the market place.  There is a difference between standard processor and semi-intelligent processor.

  18. eafpres
    January 3, 2014

    @Victor–your comment reminds me of the saying “that thing is too smart for its own good”.

  19. Netcrawl
    January 4, 2014

    @Daej I think we got this in the market, with today's fast-changing computing environment, a key benefits delivered by semi-intelligent processor could be additional performance for very high demanding applications and sophisticated systems, semi-intelligent processor could also address growing performance demands and often comnsume less power.

  20. Davidled
    January 4, 2014

    I image that in the future, self-healing or self-troubleshooting would be belonged to “semi-intelligent processor. Processor would adjust the performance depending on signal-conditional circuit condition in   the self-healing. For self-troubleshooting, instead that engineer troubleshoots the processor and circuit, processor provides the guideline to engineer or fixes itself.

  21. samicksha
    January 6, 2014

    Kinect reminds me of CMOS sensors, one of the important players in the system. Ignoring its noise factor, any reviews of same.

  22. etnapowers
    January 7, 2014
    There is a really good solution IC silicon based for gesture recognition , look at this link
     
     
     
  23. Brad Albing
    January 7, 2014

    @etnapowers — thanks for that link. Look for another Planet Analog blog that touches on aspects of gesture recognition. That can be seen here:

    The Internet of Things, Gesture Recognition & Robodog ERIC

    The Internet of Things, Gesture Recognition & Robodog ERIC

  24. etnapowers
    January 8, 2014

    Thank you Brad, The gesture recognition is an important part of the Robodog dual core solution, that is a very good example of the IoT approach.

  25. PCR
    January 15, 2014

    Bradd thanks for the timely valuable article and the links, when it comes to gesture recognitions I believe that 20cm is much more reasonable distance cause that if it recognize large distance there will be a great possibility of errors. 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.