I was recently involved in a lot of Image Processing for a project at college and while I’m no longer interested in what I was working on, I still wanted to do something fun with whatever OpenCV and Image Processing Techniques I had learnt.
I imagined an “On Screen Drums Application” where we use a live video input to detect a “drumstick” we hold up and use to beat on a virtual drum causing it to reverberate depending on how hard we hit it.
Its not too complex and can quite easily be done with some simple OpenCV tricks in Java.
First we receive the video stream and process it frame by frame. We filter each frame for the specific color and shape of the marker on our “drumstick”. In my case, my drumstick is a bright orange ball stuck to a pencil. Once we’ve identified where the marker is, we compare its current position with its position in the previous frame to roughly calculate how fast it’s traveling.
We overlay an image of a drum on the video feed and when the marker touches the overlay, we use Java’s inbuilt MIDI synthesizers to generate drum noises depending on the speed at which the drum overlay was struck.
Identifying the marker :
We first convert the frame to HSV as that color palette is much easier to compute with, rather than RGB.
To identify the marker on the drumstick, start by filtering out its hsv value. A hsv range of (100, 100, 100) to (112, 255, 255) generally captures my marker alone and leaves out most other things. Anything outside that threshold gets filtered out and that leaves us with just the marker. We run a couple of erode and dilate methods through the filtered image and that leaves us with a blob which is our marker in the frame. We also know the general size of the ball so getting the third moment of the blob would give its size. Checking if the moment is always larger than a fixed number would further eliminate mistakes.
Drawing the Drum overlay:
We use Java’s Graphics class to draw a drum overlay on the video feed. The overlay is an image of a drum cropped down to size and the rest of the screen filled with transparency. It’s saved in png format to preserve transparency.
Generating Drum Beats:
We initialize Java’s MIDI synthesizers and select the channel which sounds best. I liked channel 114 and that’s what I’ve used. I honestly can’t tell the difference between most channels, so it makes no difference to me. We check if the position of the marker intersects with the overlay on screen and if its velocity is greater than a suitable threshold. If so, then we tell Java to bang its virtual drums as hard as we virtually hit it.
The whole project is on my GitHub page at
https://github.com/Vetox/Virtual-On-Screen-Drum
Here are a few screenshots :
The marker on screen with the overlay
The same frame’s HSV image:
The same frame after filtering:
Karthi
Karthikeyan M : Geek, Gamer, Coder, Blogger - he shares his love of Technology in the form of tutorials, reviews, and how-to’s. An avid reader, he is a student at SRM university pursuing his B.Tech. in Information Technology.
