In programming a “hook” is a way for a programmer to access already existing code and modify it to his specification. A mouse hook in java is quite similar to a MouseListener, except while a MouseListener functions only within the swing or awt component it is bound to, a native mouse hook can intercept mouse calls from all over the system. This makes it immensely useful for various kinds of programs. Here, I’ll try to explain how the jnativehook library can be used to detect mouse and keyboard calls.

jnativehook is an opensource native hook library for Java. It functions as a Global Mouse and Keyboard Listener.

jnativehook can be downloaded from its official GitHub page here :
https://github.com/kwhat/jnativehook/releases
Extract and add jnativehook to your build path.

Mouse Hook :
To use a Mouse Hook in your program, have your class implement NativeMouseInputListener.
Methods to override :
public void nativeMouseClicked(NativeKeyEvent e);
public void nativeMousePressed(NativeKeyEvent e);
public void nativeMouseReleased(NativeKeyEvent e);
public void nativeMouseMoved(NativeKeyEvent e);
public void nativeMouseDragged(NativeKeyEvent e);


These methods are self explanatory. Put the code you want to run when your mouse is clicked, pressed, released, moved or dragged in these methods.

To register jnativehook and cause it to start listening to mouse events globally, enclose this line in a try catch block :
GlobalScreen.registerNativeHook();


To unregister it :
GlobalScreen.unregisterNativeHook();


For a program which detects and outputs which mouse actions have been performed :

import org.jnativehook.GlobalScreen;
import org.jnativehook.NativeHookException;
import org.jnativehook.mouse.NativeMouseEvent;
import org.jnativehook.mouse.NativeMouseInputListener;

// Add the appropriate listeners.
GlobalScreen.addNativeMouseListener(example);
GlobalScreen.addNativeMouseMotionListener(example);
}
}

Similarly for a Keyboard Hook have your class implement NativeKeyListener.
Methods to override :

public void nativeKeyPressed(NativeKeyEvent e)
public void nativeKeyReleased(NativeKeyEvent e)
public void nativeKeyTyped(NativeKeyEvent e)


The method NativeKeyEvent.getKeyText(e.getKeyCode()) will return the key that was pressed. Where e is the NativeKeyEvent accepted by each of the overriden methods.


import org.jnativehook.GlobalScreen;
import org.jnativehook.NativeHookException;
import org.jnativehook.keyboard.NativeKeyEvent;
import org.jnativehook.keyboard.NativeKeyListener;

GlobalScreen.addNativeKeyListener(new GlobalKeyListenerExample());
}
}


Overriding the nativeKeyPressed() method like above will output the KeyText of every button pressed. It will also unregister the nativehook if the “ESC” key is pressed.

I was recently involved in a lot of Image Processing for a project at college and while I’m no longer interested in what I was working on, I still wanted to do something fun with whatever OpenCV and Image Processing Techniques I had learnt.

I imagined an “On Screen Drums Application” where we use a live video input to detect a “drumstick” we hold up and use to beat on a virtual drum causing it to reverberate depending on how hard we hit it.

Its not too complex and can quite easily be done with some simple OpenCV tricks in Java.

First we receive the video stream and process it frame by frame. We filter each frame for the specific color and shape of the marker on our “drumstick”. In my case, my drumstick is a bright orange ball stuck to a pencil. Once we’ve identified where the marker is, we compare its current position with its position in the previous frame to roughly calculate how fast it’s traveling.

We overlay an image of a drum on the video feed and when the marker touches the overlay, we use Java’s inbuilt MIDI synthesizers to generate drum noises depending on the speed at which the drum overlay was struck.

Identifying the marker :
We first convert the frame to HSV as that color palette is much easier to compute with, rather than RGB.

To identify the marker on the drumstick, start by filtering out its hsv value. A hsv range of (100, 100, 100) to (112, 255, 255) generally captures my marker alone and leaves out most other things.  Anything outside that threshold gets filtered out and that leaves us with just the marker. We run a couple of erode and dilate methods through the filtered image and that leaves us with a blob which is our marker in the frame. We also know the general size of the ball so getting the third moment of the blob would give its size. Checking if the moment is always larger than a fixed number would further eliminate mistakes.

Drawing the Drum overlay:
We use Java’s Graphics class to draw a drum overlay on the video feed. The overlay is an image of a drum cropped down to size and the rest of the screen filled with transparency. It’s saved in png format to preserve transparency.

Generating Drum Beats:
We initialize Java’s MIDI synthesizers and select the channel which sounds best. I liked channel 114 and that’s what I’ve used. I honestly can’t tell the difference between most channels, so it makes no difference to me. We check if the position of the marker intersects with the overlay on screen and if its velocity is greater than a suitable threshold. If so, then we tell Java to bang its virtual drums as hard as we virtually hit it.

The whole project is on my GitHub page at
https://github.com/Vetox/Virtual-On-Screen-Drum

Here are a few screenshots :
The marker on screen with the overlay

The same frame’s HSV image:

The same frame after filtering: