My Simple HACK- Displaying fruit nutritional information using MAKEY MAKEY

Hello,

So I have been busy designing a simple hack, though I enjoyed it a lot and learned many things.

This hack will display the image of a fruit, when a fruit is touched.

Now, wait, I did not use any sensor and all, the only thing which I did is that I mapped the fruits with keys, using python, and when that fruit is touched, which is equivalent to a key press, the jpg format gets uploaded, which looks as if by touch the fruit, image of that fruit is getting displayed.

Full explanation:

Let me explain, to begin with, What is MAKEY-MAKEY:




MaKey MaKey is an invention kit for the 21st century. Turn everyday objects into touchpads and combine them with the internet. It's a simple Invention Kit for Beginners and Experts doing art, engineering, and everything inbetween:

It comes ready to use out of the box with everything you see above: MaKey MaKey, Alligator Clips, USB Cable.

How Does it Work?

Alligator Clip two objects to the MaKey MaKey board. For example, you and an apple.

When you touch the apple, you make a connection, and MaKey MaKey sends the computer a keyboard message. The computer just thinks MaKey MaKey is a regular keyboard (or mouse). Therefore it works with all programs and webpages, because all programs and webpages take keyboard and mouse input. 

Make + Key = MaKey MaKey! 

Who is MaKey MaKey For?

Artists, Kids, Educators, Engineers, Designers, Inventors, Makers... Really it is for everyone.

What materials work with MaKey Makey?

Any material that can conduct at least a tiny bit of electricity will work (if it doesn't already work, just rub it with bananas, spray it with water, or apply copper tape). Here are some materials people have used in our workshops including Ketchup, Pencil Graphite, Finger Paint, Lemons, etc.:

Other materials that work great: Plants, Coins, Your Grandma, Silverware, Anything that is Wet, Most Foods, Cats and Dogs, Aluminum Foil, Rain, and hundreds more...

Wait... Is this thing an Arduino?

You could say this board is 2 for the price of 1. MaKey MaKey runs on top of Arduino. You can start using your MaKey MaKey board in "Arduino mode" at any time. This would allow you to spin motors, turn on LEDs, or anything else that an Arduino can do. If you want to learn to use Arduino or other electronics, but want to start without any programming or breadboarding, MaKey MaKey is a good starting point. There's no need to understand Arduino in order to use MaKey MaKey.

What is PYTHON:

Python is a widely used general-purpose, high-level programming language. Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than would be possible in languages such as C. The language provides constructs intended to enable clear programs on both a small and large scale.

Python supports multiple programming paradigms, including object-oriented, imperative and functional programming or procedural styles. It features a dynamic type system and automatic memory management and has a large and comprehensive standard library.[21]

Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts. Using third-party tools, such as Py2exe, or Pyinstaller,[22] Python code can be packaged into standalone executable programs. Python interpreters are available for many operating systems.

CPython, the reference implementation of Python, is free and open source software and has a community-based development model, as do nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.

Python is a multi-paradigm programming language: object-oriented programming and structured programming are fully supported, and there are a number of language features which support functional programming and aspect-oriented programming (including by meta programming  and by magic methods). Many other paradigms are supported using extensions, including design by contract and logic programming.

Python uses dynamic typing and a combination of reference counting and a cycle-detecting garbage collector for memory management. An important feature of Python is dynamic name resolution (late binding), which binds method and variable names during program execution.

The design of Python offers only limited support for functional programming in the Lisp tradition. The language has map(), reduce() and filter() functions, comprehensions for lists, dictionaries, and sets, as well as generator expressions.The standard library has two modules (itertools and functools) that implement functional tools borrowed from Haskell and Standard ML.

Packages Used:

The python packages used to develop this interfaces are:

• PySerial : A module which  encapsulates the access for the serial port. It provides backends for Python running on Windows, Linux, BSD (possibly any POSIX compliant system), Jython and IronPython (.NET and Mono). The module named “serial” automatically selects the appropriate backend.

Features:

• Same class based interface on all supported platforms.
• Access to the port settings through Python properties.
• Support for different byte sizes, stop bits, parity and flow control with RTS/CTS and/or Xon/Xoff.
• Working with or without receive timeout.
• File like API with “read” and “write” (“readline” etc. also supported).
• The files in this package are 100% pure Python.
• The port is set up for binary transmission. No NULL byte stripping, CR-LF translation etc. (which are many times enabled for POSIX.) This makes this module universally useful.
• Compatible with io library (Python 2.6+)
• RFC 2217 client (experimental), server provided in the examples.

• OpenCV- Python is a library of Python bindings designed to solve computer vision problems.OpenCV-Python makes use of Numpy, which is a highly optimized library for numerical operations with a MATLAB-style syntax. All the OpenCV array structures are converted to and from Numpy arrays. This also makes it easier to integrate with other libraries that use Numpy such as SciPy and Matplotlib.

Other packages used are os and sys.

The important packages in this python script are serial and the opencv.

The code:

How this works:

The MaKey MaKey device is connected to the serial port. The function serial which is imported using the pyserial package reads and detect this device at a particular baud rate.  

The serial port at which the device is read is dependent on the operating system.

Currently, the python script captures the win32, linux2 and cygwin platform.

For win32, the serial port has to be configured  by using the serial instance 'COM%', for linux2, the '/dev/ttyACM0/' is the serial instance, and for cygwin, it is 'COM%+1'.

Python program reads the device at this serial port, and waits for the command by listening to the 'COM' port.

The MaKey MaKey device used in this case is a deluxe kit. The deluxe kit has 6 Keyboard, which acts as a normal computer keyboard.

Connect the fruit orange to the key 'W' , apple to 'A', and banana to 'S'. Connect any other fruit to 'D'

Here, the package opencv is used to display the image. This package comes with a method cv2.waitKey() which waits for the key press. Use 0 so that the program waits for the inifnite amount of time for the key to be pressed.

The file is kept in the directory, which is read using cv2.imread() function, if file is not found at that location, then it raises an error.

The opencv uses the ASCII character of the key which has been pressed, so, even if 'w' is pressed, the ascii value of 'w' which is 119,will be used to process further.

The image loaded using the cv2.imshow() function,in which image which has been read from the directory. 

In order to make the program in such a fashion, that each time, the image is loaded freshly, the cv2.destroyAllWindows() function is used, which destroys the previous loaded window and loads the window afresh. 

So when the fruit orange is pressed, which is connected to the key 'w', whose ascii value is ready by the serial port, the file orange.jpg which is present in the directory gets loaded, and the image of the orange with its nutritional value appears. In order to fit the image properly, the resize function is used.

To exit the program, the ascii value of space is used, touch the space key of the MaKey MaKey, which closes the serial device and the program gets terminated, returning the control back to the python interpreter.

The circuit:




Connect the Makey Makey to the circuit, if the device is not connected, then there is an exception raised which is: 

Now, after the circuit is connected, it will load a 'welcome.jpg' image,as in the program the imshow() function encounters this image: 

When we touch orange, it is equivalent to the key press 'w', the waitkey(0) function waits for the key, destroys the previous window, and the imshow window is set to show the orange.jpg file:

When we touch apple, it is equivalent to the key press 'a', the waitkey(0) function waits for the key, destroys the previous window, and the imshow window is set to show the apple.jpg file:

Similarly with the banana.

when we touch the space button, the program exits.

Potential issues faced:

•  The program does not work on cygwin, until and unless the win32 does not read the device. Even if the device is connected, and program is executed using cygwin, it still raises an issue 'Device not found'. I researched on this further, and found that cygwin does not detect virtual COM ports, and despite, giving the mapping of the device, which is present in the serialposix.py, device is not being detected, if the program is not executed in win32. 

• For linux 2, the port /dev/ttyACM0/ has to be configured using dmesg or gtkterm command. The baud rate mismatch also makes the device undetectable.

• I found an interesting thing with python list. In a python list, an element can be accessed using the index, but if we try to access the element using a for loop, then it fails. The reason being that in a for loop, the python list elements or indexes have to be fetched in the sequential order, which means, that the elements will be fetched in the order at which they are placed in the list. Trying to access a list, which is not equal to it's index number, while the program is in for loop will raise an error 'Element referenced before an assignment' , though the same element can be accessed if the index number is known, without for loop, as in this case, order does not matter.

• If the fruit is cold, then the response time when a fruit is pressed and image gets uploaded is very low, as if, it happens instantaneously. But, if the fruit is at room temperature, then, a bit of force has to be applied depending on the hardness of the fruit, in order to get the image uploaded. This happens especially with apple, which has to be pressed using the palm also, to get the image uploaded instantaneously.

• The circuit must not be kept on a surface which is slippery, like floor marble, rather, use wooden table, so that, until and unless, person does not touch the fruit, the keys don't get detected.

Further Hacks:

• One of the thing which I was trying to achieve was to display the image in a web browser, by pressing the keys. But the problem here is that the browser window and the window key press events of opencv2 are mutually exclusive, and hence, the key press event  waitKey(0)  is not being detected by the browser window. I am still trying to figure out a way to resolve this issue.

•   Another thing which I need to sort out is that irrespective of which fruit is connected to a particular key, the image of that fruit must get uploaded. At present, I am hard coding the key values and also, the fruit images, and connecting the fruits to these keys. My intention is that, irrespective of any fruit  connected to any key, the keys must automatically detect the fruit which is connected, and image or information of that fruit must get displayed. This can be done by converting the physical information or property of a particular fruit into digital format, and getting this digital information read, and converting it into file. This is known as Physical Computing, where we turn 'THINGS' into 'DATA' and 'DATA' into 'THINGS'. Here, the physical fruit consists of atoms and its digital part or information, is bit. So we turn the physical property, atom, into bits, or data, which is digital information, and get the information out of it. Similarly, reverse approach can be take, means, using the digital information of bits, we can get that physical object which this digital file represents. Though, need to figure out, the role of keys here, as this hack is just for fun.