Time-frequency signal representation for polyphonic music

We introduce a search-based approach for obtaining musically valid time-frequency (TF) representations of polyphonic music. The plethora of available tradeoffs between time and frequency resolution at each point in the TF plane leads to a combinatorial explosion in the search space of possible representations. The musical validity of each TF representation in this space is highly signal dependent. To account for this, we utilize constraint matching for identifying and subsequently guiding the use of signal reprocessing to repair invalid subregions of an initially selected TF representation. In contrast to existing data-adaptive techniques for TF signal representation, our approach exploits knowledge of non-local as well as local domain constraints at multiple levels of abstraction in the TF plane. To enable such multi-level constraint matching in conjunction with sophisticated signal reprocessing capabilities, we have used the Integrated Processing and Understanding of Signals (IPUS) framework as the supporting system architecture. Such IPUS-based TF analysis is also potentially useful for non-musical applications in which domain constraints at multiple levels of abstraction can aid the search for valid TF representations.;In the musical domain, our approach provides significant resolution improvement with respect to the constant-Q representation. The use in our approach of short-term musical constraints such as those on harmony, vibrato, and tremolo is also shown to provide at least as much resolution improvement as that offered by the Adaptive Time-Frequency Representation (ATFR) of Parks and Jones. Experimental validation of this improvement has been carried out through the analysis of a variety of Musical Instrument Digital Interface (MIDI) musical signals on a software system realization of our approach. It should also be noted that the computational cost of our approach is guaranteed to be at least an order of magnitude less than the cost of the ATFR. Furthermore, in contrast to the ATFR, our approach holds the potential of further resolution improvement through the inclusion of longer-term constraints on melody and rhythm.;In many applications, the TF analysis of polyphonic music is a precursor to the detection of onset and offset transients. We have developed a transient detection technique that can be directly applied to the outputs obtained from our approach to TF analysis. It is shown that such detection is statistically superior to the traditional derivative-based approach. For example, when the tone-to-noise ratio is 0 dB and the probability of false alarm is held at 0.05, there is an approximately 85% improvement in probability of detection at a point half a time-constant away from the maximum transient amplitude. Furthermore, the percentage improvement in the probability of detection increases with decreasing amplitudes in the transient region.