My project is a handpose based instrument which has both beats and synth, along with a tutorial which teaches you to play the instrument.
https://github.com/eggwaffles/handpose-instrument
This is inspired by instruments such as the theremin and other interactive instruments I’ve seen. Last semester I worked on a project connecting the kinect camera to TouchDesigner to control instruments in Ableton so I’m wondering if it’s possible to do this with the webcam and HandPose so it’s more accessible for users to play just from a website, without needing a lot of software and equipment.
I started with one of the handpose models from class to start with a sketch with handpose set up. I then added the p5 sound library so I could add sounds directly into the code.
For each gesture that needed to be tracked, I wrote a function which measured the positions of different handpose points. For example this is the function to determine whether or not I pinched my right hand or not. I wrote this function to detect whether the palm was open or closed, as well as the number of fingers that were raised (although I didn’t end up using this in the final project)
function rightisPinched() {
for (let i = 0; i < hands.length; i++) {
let hand = hands[i];
if (hand.handedness === "Left") {
let thumbTip = hand.keypoints.find(point => point.name === "thumb_tip");
let indexTip = hand.keypoints.find(point => point.name === "index_finger_tip");
if (thumbTip && indexTip) {
let distance = dist(thumbTip.x, thumbTip.y, indexTip.x, indexTip.y);
if (distance < 30) {
return true;
}
}
}
}
return false;
}
I wanted the left hand’s gestures to act as an instrument selector. I originally envisioned this project to include 5 instruments, but I ended up only having the time to create 2. Because of this, synths used to be triggered by holding up 1 finger and drums were triggered by holding up 2.
However, during the feedback session I noticed that people often started interacting with the instrument by holding both their hands up with their palms open, which didn’t produce any sound because there is no instrument mapped to 5 fingers up. So Instead of holding up 1 and 2 fingers, I decided to change it to hand open and closed to select between drums and synths. This created immediate feedback for the user when they walked up to try the instrument.