Contents
This textbook provides both profound technological knowledge and a comprehensive treatment of essential topics in music processing and music information retrieval (MIR). Including numerous examples, figures, and exercises, this book is suited for students, lecturers, and researchers working in audio engineering, computer science, multimedia, and musicology. In the following, we give an overview of the book's content. The preface of the book contains further information on the overall structure of the book, the interconnections between the various topics and techniques, as well as suggestions on how this book may be used as a basis for different courses. A PDF of the table of content (Springer 2021) can be obtained here.
Overview
This textbook consists of eight chapters. The first two chapters cover fundamental material on music representations and the Fourier transform—concepts that are required throughout the book. These two chapters make the book self-contained to a great extent. In the subsequent chapters, concrete music processing tasks serve as starting points for our investigations. Each of these chapters is organized in a similar fashion. A chapter starts with a general description of the music processing scenario at hand and integrates the topic into a wider context. Motivated by the scenario at hand, each chapter discusses important techniques and algorithms that are generally applicable to a wide range of analysis, classification, and retrieval problems. All these techniques are treated in a mathematically rigorous way. At the same time, the techniques are immediately applied to a concrete music processing task. By mixing theory and practice, the book's goal is to convey both profound technological knowledge as well as a solid understanding of music processing applications. At the end of each chapter, we provide links to the research literature, hints for further reading, a list of references, and exercises.
Furthermore, in the book's second edition (Springer 2021), we provide at the end of each chapter an additional section titled FMP notebooks. These sections serve two purposes. First, we give a comprehensive guide by systematically describing the content and purpose of all the notebooks related to the corresponding chapter. As a second objective, we make concrete suggestions on using the FMP notebooks to create an enriching, interactive, and interdisciplinary supplement in the form of experiments and advanced studies in a music processing curriculum. The textbook's guide can be best appreciated and understood when the FMP notebooks run in a browser simultaneously while reading.
We now give an overview of the individual chapters and the main topics.
Chapter 1: Music Representations
Musical information can be represented in many different ways. In Chapter 1, we consider three widely used music representations: sheet music, symbolic, and audio representations. This first chapter also introduces basic terminology that is used throughout the book. In particular, we discuss musical and acoustic properties of audio signals including aspects such as frequency, pitch, dynamics, and timbre.
- Sheet Music Representations
- Symbolic Representations
- Audio Representation
- Summary and Further Readings
- FMP Notebooks
Chapter 2: Fourier Analysis of Signals
Important technical terminology is covered in Chapter 2. In particular, we approach the Fourier transform—which is perhaps the most fundamental tool in signal processing—from various perspectives. For the reader who is more interested in the musical aspects of the book, Section 2.1 provides a summary of the most important facts on the Fourier transform. In particular, the notion of a spectrogram, which yields a time–frequency representation of an audio signal, is introduced. The remainder of the chapter treats the Fourier transform in greater mathematical depth and also includes the fast Fourier transform (FFT)—an algorithm of great beauty and high practical relevance.
- The Fourier Transform in a Nutshell
- Signals and Signal Spaces
- Fourier Transform
- Discrete Fourier Transform (DFT)
- Short-Time Fourier Transform (STFT)
- Summary and Further Readings
- FMP Notebooks
Chapter 3: Music Synchronization
As a first music processing task, we study in Chapter 3 the problem of music synchronization. The objective is to temporally align compatible representations of the same piece of music. Considering this scenario, we explain the need for musically informed audio features. In particular, we introduce the concept of chroma-based music features, which capture properties that are related to harmony and melody. Furthermore, we study an alignment technique known as dynamic time warping (DTW), a concept that is applicable for the analysis of general time series. For its efficient computation, we discuss an algorithm based on dynamic programming—a widely used method for solving a complex problem by breaking it down into a collection of simpler subproblems.
- Audio Features
- Dynamic Time Warping
- Applications
- Summary and Further Readings
- FMP Notebooks
Chapter 4: Music Structure Analysis
In Chapter 4, we address a central and well-researched area within MIR known as music structure analysis. Given a music recording, the objective is to identify important structural elements and to temporally segment the recording according to these elements. Within this scenario, we discuss fundamental segmentation principles based on repetitions, homogeneity, and novelty—principles that also apply to other types of multimedia beyond music. As an important technical tool, we study in detail the concept of self-similarity matrices and discuss their structural properties. Finally, we briefly touch the topic of evaluation, introducing the notions of precision, recall, and F-measure. These measures are used to compare the computed results that are obtained by an automated procedure with so-called ground truth annotations that are typically generated manually by some domain expert.
- General Principles
- Self-Similarity Matrices
- Audio Thumbnailing
- Novelty-Based Segmentation
- Evaluation
- Summary and Further Readings
- FMP Notebooks
Chapter 5: Chord Recognition
In Chapter 5, we consider the problem of analyzing harmonic properties of a piece of music by determining a descriptive progression of chords from a given audio recording. We take this opportunity to first discuss some basic theory of harmony including concepts such as intervals, chords, and scales. Then, motivated by the automated chord recognition scenario, we introduce template-based matching procedures and hidden Markov models—a concept of central importance for the analysis of temporal patterns in time-dependent data streams including speech, gestures, and music.
- Basic Theory of Harmony
- Template-Based Chord Recognition
- HMM-Based Chord Recognition
- Summary and Further Readings
- FMP Notebooks
Chapter 6: Tempo and Beat Tracking
Tempo and beat are further fundamental properties of music. In Chapter 6, we introduce the basic ideas on how to extract tempo-related information from audio recordings. In this scenario, a first challenge is to locate note onset information—a task that requires methods for detecting changes in energy and spectral content. To derive tempo and beat information, note onset candidates are then analyzed with regard to quasiperiodic patterns. This leads us to the study of general methods for local periodicity analysis of time series.
- Onset Detection
- Tempo Analysis
- Beat and Pulse Tracking
- Summary and Further Readings
- FMP Notebooks
Chapter 7: Content-Based Audio Retrieval
One important topic in information retrieval is concerned with the development of search engines that enable users to explore music collections in a flexible and intuitive way. In Chapter 7, we discuss audio retrieval strategies that follow the query-by-example paradigm: given an audio query, the task is to retrieve all documents that are somehow similar or related to the query. Starting with audio identification, a technique used in many commercial applications such as Shazam, we study various retrieval strategies to handle different degrees of similarity. Furthermore, considering efficiency issues, we discuss fundamental indexing techniques based on inverted lists—a concept originally used in text retrieval.
- Audio Identification
- Audio Matching
- Version Identification
- Summary and Further Readings
- FMP Notebooks
Chapter 8: Musically Informed Audio Decomposition
In the final Chapter 8 on audio decomposition, we present a challenging research direction that is closely related to source separation. Within this wide research area, we consider three subproblems: harmonic–percussive separation, main melody extraction, and score-informed audio decomposition. Within these scenarios, we discuss a number of key techniques including instantaneous frequency estimation, fundamental frequency (F0) estimation, spectrogram inversion, and nonnegative matrix factorization (NMF). Furthermore, we encounter a number of acoustic and musical properties of audio recordings that have been introduced and discussed in previous chapters, which rounds off the book.
- Harmonic-Percussive Separation
- Melody Extraction
- NMF-Based Audio Decomposition
- Summary and Further Readings
- FMP Notebooks