Advertisement

Blog

The Internet of Things, Gesture Recognition & Robodog ERIC

Recently my company STMicroelectronics has joined the ARM mbed Project that is a project for companies interested in producing integrated SoC by utilizing an IoT (Internet of Things) approach.

The main goal of this project (according to some promotional material) is “to meet the needs of a new professional developer audience. It delivers free tools and fundamental open-source hardware and software building blocks for the rapid development of innovative ARM-based devices. The project also enables the easy integration of connectivity, sensor, and cloud-service software components and the tools and support for a dynamic, collaborative developer and partner ecosystem.”

The director of IoT platform development, Simon Ford at ARM, said: “The mbed project is bringing together leading technology companies to create a step change in productivity for embedded device development. We have learnt from the web and smartphone revolutions that by building an open-source software platform with reusable software components and free development and collaboration tools, we can enable the creation of IoT and smart devices on a previously unimagined scale.” (Source: ST.com Press Release.)

The community of system and software engineers has a dedicated website at ARM to help the developers collaborate. This helps them discuss their experiences in the field of the production of embedded systems. For example, implementing complex functions, such as image processing and gesture recognition.

The image processing function is featured by one of the partners of the community: the eyeSight company. I found very interesting the solutions section of the company website: “Using eyeSightEmbedded our machine vision algorithms can be implemented on DSP and GPU level, allowing the technology to be distributed between different processing units, offering a very powerful and efficient gesture recognition system without impacting the main application processor.”

The technology used in eyeSight's Touch Free system “utilizes advanced real-time image processing and machine vision algorithms to track the user's hand gestures and convert them into commands. [See Figure 1.] These commands are then used to control functions and applications within the device, creating a Natural User Interaction. The technology is completely software based and is independent of the underlying processor and camera hardware. The technology from eyeSight produces robust gesture recognition using only a standard 2D camera. It is compatible with 3D stereoscopic sensors, and IR illumination. The Touch Free technology can be easily integrated into various levels of the digital device: on the chipset level, operating system, as part of the camera module, or simply integrated in application level.”

Figure 1

eyeSight's Touch Free technology virtual mouse solution. The object shown on the screen inside the shop can be selected from outside by a gesture.

eyeSight's Touch Free technology virtual mouse solution. The object shown on the screen inside the shop can be selected from outside by a gesture.

The technology is really optimized in terms of power consumption of the CPU. This makes this solution ideal for integrated SoC systems. It is based on a standard VGA camera. Moreover, it is suitable for handheld devices like smartphones and tablets. The user can easily control many devices (such as smartphones, tablets, PCs, and set-top-boxes) in a touch-free mode. This feature also can be utilized for the movement detection because the visual recognition eyeSight technology can also operate in a wide range of lighting conditions, even if they are very dynamic and fast-changing. (See Figure 2.)

Figure 2

eyeSight's Touch Free technology overview.

eyeSight's Touch Free technology overview.

The features of this technology makes it very suitable to realize a complex system based on an Internet of Things IoT approach, and, following this philosophy, the ARM community presented the Robodog solution in a video that shows all the potentialities of this approach. The Robodog ERIC (Embedded Robotic Interactive Canine) incorporates concepts from:

  • Electronics engineering
  • Mechanical engineering
  • Software engineering
  • Computer engineering
  • Inverse kinematics
  • Speech recognition

The ERIC system is able to recognize a specific object by a visual identification. It is implemented by means of a touch-free solution (as with the eyesight system). The Robodog can move to the selected object and grab it among a scenario of other different objects. (See Figure 3.)

Figure 3

The ERIC Robodog chooses an object among a scenario of three possible objects and moves to grab it.

The ERIC Robodog chooses an object among a scenario of three possible objects and moves to grab it.

The Robodog Eric central core is implemented with two microcontrollers: The first is dedicated to the movement control and the second is for the image processing. (See Figure 4.)

Figure 4

ERIC Robodog's dual core processors.

ERIC Robodog's dual core processors.

This dual core solution is open to be adapted to the requirements of the company which decides to join the ARM mbed project and that is interested in the production of smart integrated systems controlled by microcontrollers acting as the CPU that realizes image processing, actuator procedure, motor control, by mean of a IoT approach. The Robodog may be utilized also to guard an object and to react when the selected object is taken away, and it recognizes a verbal command.

Have you ever worked with an embedded SoC similar to the ERIC Robodog, presented in this blog which integrates an image processing and motor control procedure? What do you think about the potential usage of this system? Did you ever use a dual core microprocessor approach to realize an IoT embedded system? What do you think of eyeSight's Touch Free technology?

Related posts:

23 comments on “The Internet of Things, Gesture Recognition & Robodog ERIC

  1. etnapowers
    January 8, 2014

    The double core solution here proposed for the robodog is really smart because it provides high computing rate, provided that a synchronization of the two microprocessors is effective.

  2. etnapowers
    January 8, 2014

    I guess the dual core strategy is a really good solution, because the master microprocessor can control the slave and it performs additional controls of movement but when it is needed the roles can be inverted , being the image processor micro-controller the master and the other one being the slave.

  3. samicksha
    January 8, 2014

    Sounds like a new wave of technology which need a receiver device that listens and understand to my Wi-Fi transmissions.

  4. etnapowers
    January 8, 2014

    Yes Samicksha, I agree with you, that's an approach based  on the internet of thing strategy. this technology holds promises of high diffusion in the near future.

  5. Victor Lorenzo
    January 8, 2014

    I find very interesting the line of research on distributed asymmetric multiprocessor systems for robotic control applications. I imagine a system with no static master and slave controllers, but coordinated sub-systems supporting redundancy and hot-swapping, with several high-speed dynamic priority and allocation interconnections.

  6. Davidled
    January 8, 2014

    According to picture, it looks like two core processors are used in the board. I wonder what type voice recognition engine is embedded in the processor.

  7. goafrit2
    January 8, 2014

    >> provided that a synchronization of the two microprocessors is effective.

    In the industry and from experience, the #1 challenge with multicore systems has to do with the inability of software to efficiently mine the processing powers the muticores provide. In some cases, you have these many cores yielding only value of one core because the weakest link is the software. I think that is where the challenge lies.

  8. goafrit2
    January 8, 2014

    >>  because the master microprocessor can control the slave and it performs additional controls of movement but when it is needed the roles can be inverted 

    That will most likely creare a lot of redundancies and inefficiecies in the system. You cannnot rewire microprocessor that way without wasting a lot of power. For IoT, power matters. STMicroelectronics is a leader in consumer MEMS and they have many great things going for them in that space. I think they will continue to expand and lead this market.

  9. Victor Lorenzo
    January 9, 2014

    @DaeJ, this does not respond to your question, but might be usefull for you or others. A couple of years ago I read a blog about an open source robot project. I was not able to remember the name for the project but the robot's picture was still stored somewhere in my less volatile memory ;). The robot is called Q·bo and for what it seems the project is no longer active. The blog is here (http://thecorpora.com/blog/?lang=en). For continuous speech recognition they used an engine called Julius which is open source too and can be found here (http://julius.sourceforge.jp/en_index.php)

    The Julius project is still under active development but has a drawback, for japanese language the voice models are free but not for english.

    Another project, VoxForge (http://www.voxforge.org/), is involved in poviding those models for several voice recognition engines. You'll find references there to other engines and related projects.

  10. etnapowers
    January 9, 2014

    @Victor: That's very interesting to me, it's a dynamic solution to increase the speed of iteraction and the system stability, this solution is really smart.

  11. etnapowers
    January 9, 2014

    @DaeJ: I confirm that the two core processors are used in the board at the same time , the voice recognition engine is not integrated in the processor, the solution is controlled by the  dedicated microprocessor that is able to perform other functions, among which the elaboration of the data, the protection by mean of a security engine and some security functions like emergency shutdown.

  12. etnapowers
    January 9, 2014

    @goatfrit2: the STMicroelectronics has the MEMS as one of its main product and it's really interested in all the possible fields of applications of this devices, I think that the range is really promising but there are many issues to be solved , like you correctly said.

  13. Netcrawl
    January 10, 2014

    @etnapowers you're right, it also enable the creation of IoT and smart devices on an unimaginable scale, STMicroelectronics huge portfolio combined with mbed platform could provides software portability and a rapid development of new wave of electronics products.    

  14. Netcrawl
    January 10, 2014

    @etnapowers I agree with you, MEMS is an important differentiator for mobile devices manufacturer, its enables new capabilties and functionalities, this is very interesting and promising but there still some major challenges ahead, its going to takes some time.   

  15. Netcrawl
    January 10, 2014

    @goafrit2 multicore systems has been the heart of computing revolution and big iron computing, the best things about multicore systems is that they can operate at lower frequency, consuming less power and enable high performance computing. 

    @goafrit2 you're right about challenges with multicore systems, software is a key issue here, something that need too much attention.

  16. RedDerek
    January 11, 2014

    I agree that having a dual processor helps. I remember working on a missile flyout simulator back in the early '90s. One processor simulated the missle electronics input and output. And a third processor to fly the missle and create the jamming sequence. All running on a 32 bit backplane and sharing the same hard drive and using a common memory area. Everything had to run fast and in real-time. Now this was three 80486 processor cards and was fit in a 19″ rack unit. Now it could be much, much smaller as we see with the RoboDog. 

    I also remember reading about the self driving cars about 20 years ago that had three or four computers stuffed in the trunk to perform all the imaging processing and control.

  17. fasmicro
    January 13, 2014

    >> the STMicroelectronics has the MEMS as one of its main product and it's really interested in all the possible fields of applications of this devices, I

    If nanotechnoogy has deliverred based on the hypes on the technology iabout a decade ago, we would have seen more improvements in some of the problems we encounter in MEMS. The challenge is that big funded research that would have unlocked the opportunities in NANO is rare because many politicians think that govt getting itself away from research will save America money. And some VCs that see 5 years as eternity in their internet companies will step up and fund a future they think is far.

  18. goafrit2
    January 13, 2014

    >> this is very interesting and promising but there still some major challenges ahead, its going to takes some time.

    What are those challenges. Can you mention one. The level of integration we have with MEMS now in the mobile business space is robust. Fro gyro to XL, you can see these products driving new areas no one had imagined. Sure, there are constraints in performance limited by process-technology, but I will like to know what you think are the other challenges.

  19. goafrit2
    January 13, 2014

    >> @goafrit2 you're right about challenges with multicore systems, software is a key issue here, something that need too much attention.

    And it is one few are interested to focus on. I do think we may need the synergy in the mobile app space where those that make hardware may not be specifically masters in the software. Mutilcore systems are not advancing fast because the software side is a weak link. Maybe, they can open it up for companies to figure how to program their owns for their specific uses

  20. yalanand
    March 31, 2014

     I do think we may need the synergy in the mobile app space where those that make hardware may not be specifically masters in the software.

    @goafrit2, I agree with you. But offlate many software companies are trying to build their own hardware. For example even google bought Motorola so that it could manufacture its own handsets.

  21. yalanand
    March 31, 2014

    If nanotechnoogy has deliverred based on the hypes on the technology iabout a decade ago, we would have seen more improvements in some of the problems we encounter in MEMS. 

    @fasmicro, I agree with you. But due to slow down in semiconductor industry many of the research projects got stopped but slowly things are picking up and we might see more improvements in this field.

  22. fasmicro
    April 7, 2014

     But offlate many software companies are trying to build their own hardware.

    The difference is in the details. Henry Ford “built” a car by paying people to do so. Until I see a software company organically develop a hardware business, I think they are playing Wall Street. Sure, Google has MOT but does not make it the same as HP.

  23. fasmicro
    April 7, 2014

     But due to slow down in semiconductor industry many of the research projects got stopped

    I think the problem is not slowdown in research. I think the problem is that investors found something more exciting in web and apps. For all its innvations, Analog Devices is not a $16B business but WhatsApp made it without the CFO breaking his head.

     

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.