Last updated: 2022-06-07
What’s inside
The core of the instrument is a combination of two applications running simultaneous on my computer: an existing (commercial) digital audio workstation (DAW) like Ableton Live or Logic Pro, and my own new software.
My software detects motion or moving objects and can convert the information in MIDI messages that are sent to a MIDI device that knows how to handle these messages: convert them to perceptible sounds. This can be an external synthesizer (“keyboard”) or a DAW. The sounds generated can be of any kind like piano, drums, synthesizers, but also samples of existing audio recordings. For this instrument I prefer using Ableton Live because it is very flexible in manipulating my own sound samples. Nearly all controls (knobs, dials, etc.) in this DAW can be connected to MIDI control changes (CC messages) that are generated by my software. I can generate notes (noteOn and noteOff messages), but also manipulate panorama, reverb, volume or multiple other special effects in the DAW at the same time.
The detection screen can be split up in several regions, each one is coupled to a MIDI channel. Messages are sent independently to (maximal) 16 MIDI channels, that in the DAW have all individual settings and sounds.
My program has many detection algorithms. Some are based on simple motion detection (subtraction of two successive frames), color detection (detect objects of a adjustable color) or object tracking. Therefore the object detection algorithms use one or more rectangles in a still from the camera, and follow these. So I can use my head, nose or an eye to generate sounds.
Because most forms of detection algorithms can produce a lot of random noise, they don’t have somenthing like persistence over time, I implemented an adjustable averaging mechanism to smooth the results, the detected movements.
How to use it
My software requires a recent MacOS computer with a webcam and a MIDI device.
As camera you can use the build-in webcam or an external camera. I also implemented functionality to use the webcam of another Mac in the same WiFi network, with very low latency.
For the MIDI device you can use any DAW, software synthesizer or external synthesizer that connects to the computer via the IAC Driver or has its own MIDI port. Also the Apple midi network functionality can be used to send the midi stream to another computer (even Windows) in the same WiFi network.
First turn to your DAW and “arm” the input channels, ports, instruments, samples, effects etc.
After starting my software, it scans for connected webcams, resolutions and MIDI ports and selects defaults. It presents two windows:
– a control panel, and
– a video window.
With the settings in the control panel you can adjust a lot of settings (detection method, detection settings, view settings, global MIDI settings and individual MIDI channel settings).
The video window gives visual feedback on your movements and of course your MIDI devices do it with their audio. The video window can also be sent to another Mac in the same WiFi network (for example the same one that is used as camera).
It is possible to record the video and the audio by the DAW, both triggered by one single CC message issued from the user interface (GUI). This results in two separate files, one video file (.mp4 or .mov) from my program, and an audio file exported from the DAW. I build a separate software program (based on ffmpeg) that makes it possible to combine these exactly synchronised to make a demo video.
Here I put all these information in a scheme, for the technicians…
schematic overview of my software application
How is it built?
This instrument was first build on MacOS 10.11.6 (El Capitan) in C++ with Qt version 5.6.0., OpenCV 3.4.3, and RtMidi 3.0.0 as API for MIDI functionality.
In 2020 I had to change to a MacBook Pro. My latest version (2022) is build on MacOS 10.14.6 (Mojave), in C++ with Qt 5.12.9, OpenCV 4.5.3 with FFmpeg, DLib 19-21 and QMidi.
It contains all necessary libraries and runs even on a ‘clean’ machine.
The fact that this was only a MacOS application limited its broader applicability. Therefore I have ported it to a Windows 10 environment. I managed to ‘copy’ nearly all functionality and if there is a project suited for this application, I might give a trial version (Mac or Windows). Don’t hesitate to contact me.
I consider my software now as relatively “final”. This means that I will concentrate on the sounds and combinations of sounds for the Ableton samplers.
When I have more results I will show them on the Blog of this site.
Paul’s Project 