Design And Implementation Of Real Sense-based Real Time Gestural Composition System

In general, the public is accustomed to achieve improvised music in many ways, instead of create compositions by themselves. The reason is that professional composing skills are required to handle the current real-time composition systems. Further, these systems are difficult to operate. To tackle the two obstacles, we have designed and implemented a RealSense-based gestural real-time composition system in order to involve users without special training into the pleasure of music composition.We applied the recently delivered RealSense technology into gesture recognition. As a result, harmonic and diverse melodies can be generated by operating normally-used nature gestures. Our system consists of three modules:(1) gesture interaction. Users’ gestures are captured and recognized in real-time by 3D RealSense camera. The recognized results are mapped to scale chord inputs.(2) real-time composition. Along with the chord progressions, arpeggio melodies, walking bass lines and drum tracks are automatically generated by means of computer-aided algorithms. Firstly, harmonic and various arpeggios are generated based on a novel deep learning based algorithm combining gated neural unit(GRU) and tetrahedral context-sensitive L-system. Secondly, a contour-based bass generation algorithm is utilized for generate the global coherent walking bass.(3) Music output. Export the MIDI and audio generations out of system.In this paper, our system is capable of relieving the obstacles of novices to creating musically harmonic compositions. In addition to the assistance for novices, our system has promising potentials to be applied in improvisation, offering help for music learner and composers. We proposed an interactive approach of utilizing nature gestures to control chords for music generating, which can be extend to widely use in commercial scenarios.