Annotation Visualization
In this notebook, we introduce some libfmp-functions that can be used for visualizing various kinds of annotations as encountered in music processing. In particular, we consider single-value annotations and segment annotations that consist of pairs of values.
Conventions and Formats¶
In the FMP notebooks, we encounter various types of annotations that can be expressed as list of instances. Typically, an instance consists of one or several numerical values (e.g., time positions, durations, frequency values, pitches, and so on) and a label given as text string. In this notebook, we consider single-value annotations (one numerical value and a label) and segment annotations (two numerical values and a label). For more complex examples, we refer to FMP notebook on symbolic music formats. To write and read annotations, we use a simple file format based on on comma-separated values (CSV). Each instance corresponds to a row, and the elements (i.e., numerical values and label) of each row are separated by a semicolon (;) rather than a comma. Furthermore, in the first row, we include a header line that specifies the meaning of the columns (i.e., the elements).
To read, write, and modify CSV files, we often use the Python library pandas, which provides easy-to-use data structures and data analysis tools. Using the pandas framework, the content of a CSV file is stored in a two-dimensional tabular data structure with labeled axes (rows and columns) called DataFrame. This data structure can be accessed using attributes, see the documentation for pandas.DataFrame. For more details, we refer to the FMP notebook on the CSV format and the functions libfmp.b.read_csv and libfmp.b.write_csv. In the following, we look at some simple examples that illustrate our conventions.
Single-Value Annotations¶
We start with an annotation format, where each instance consists of a single numerical value. Such a value may refer to a time position (specified in seconds or frame indices) or to a frequency value (specified in Hertz or frequency bin indices). Furthermore, the semantics of this numerical value is further specified by a label, which consists of a text string. We encounter a typical example for such an annotation type in the context of tempo and beat tracking, where the values are time positions (given in seconds) and the labels specify the musical meaning of the time position (e.g., indicating a measure, quarter note, or eighth note position). In the following code cell, we generate a toy example of such an annotation, which serves as our running example. The data structure consists of a list of tuples, each tuple consisting of a value and label. We also show how the data structure can be converted into a DataFrame (df) and csv file.
import os
import sys
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import IPython.display as ipd
sys.path.append('..')
import libfmp.b
%matplotlib inline
measures = np.arange(0, 2.5, 1.0)
quarters = np.arange(0, 2.5, 0.5)
eighths = np.arange(0, 2.5, 0.25)
# Generate dummy annotations
ann = sorted([(n, 'measure') for n in measures] +
[(n, 'quarter') for n in quarters] +
[(n, 'eighth') for n in eighths], key=lambda x: x[0])
print('=== Dummy annotation ===')
print(ann)
# Convert to DataFrame and CSV format (as used for the FMP notebooks)
ann_df = pd.DataFrame(ann, columns=['position', 'label'])
ann_csv = ann_df.to_csv(sep=';', index=False, quoting=2)
print('\n=== Dummy annotation in CSV format ===')
print(ann_csv)
print('=== Dummy annotation stored as DataFrame and rendered as HTML ===')
ipd.display(ipd.HTML(ann_df.to_html()))
# Store and load annotation file
#fn_out = os.path.join('..', 'output', 'B', 'FMP_annotation_position_dummy.csv')
#ann_df.to_csv(fn_out, sep=';', index=False, quoting=2)
#ann_df = pd.read_csv(fn_out, sep=';')
#ann = [(position, label) for i, (position, label) in df.iterrows()]
The function libfmp.b.plot_annotation_line allows for plotting single-value annotations. In the following code cell, we visualize the annotations within a single plot, where each label type is assigned to an individual plotting style.
label_keys = {'measure': {'linewidth': 6, 'color': 'k'},
'quarter': {'linewidth': 3, 'color': 'b'},
'eighth': {'linewidth': 1, 'color': 'r'}}
fig, ax = libfmp.b.plot_annotation_line(ann, label_keys=label_keys)
ax.set_xlim([-1,3]);
Next, we visualize the annotations horizontally aligned to a matrix. This mimics the situation, where the horizontal axis of the matrix has the semantics expressed by the annotations.
matrix = np.random.random((16, 16))
fig, ax = plt.subplots(2, 1, sharex=True,
gridspec_kw={'height_ratios': [4, 1]}, figsize=(6, 3))
ax[0].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-1, 3, 0, 1], alpha=0.5)
libfmp.b.plot_annotation_line(ann, ax=ax[1], label_keys=label_keys, time_min=-1, time_max=3)
plt.tight_layout()
Similarly, the annotations can be visualized vertically aligned to the matrix.
fig, ax = plt.subplots(1, 2, sharey=True,
gridspec_kw={'width_ratios': [1, 10]}, figsize=(6, 3))
ax[1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[0, 1, -1, 3], alpha=0.5)
libfmp.b.plot_annotation_line(ann, ax=ax[0], label_keys=label_keys,
direction='vertical', time_min=-1, time_max=3)
plt.tight_layout()
Similar to libfmp.b.plot_annotation_line, there also is the function libfmp.b.plot_annotation_line_overlay, which can be used to overlay a matrix with the annotations horizontally as well as vertically.
fig, ax = plt.subplots(1, 2, gridspec_kw={'width_ratios': [3, 2]}, figsize=(8, 3))
ax[0].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-1, 3, 0, 1], alpha=0.5)
libfmp.b.plot_annotation_line_overlay(ann, ax=ax[0], label_keys=label_keys, time_min=-1, time_max=3)
ax[1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[0, 1, -1, 3], alpha=0.5)
libfmp.b.plot_annotation_line_overlay(ann, ax=ax[1], direction='vertical',
label_keys=label_keys, time_min=-1, time_max=3)
plt.tight_layout()
The function libfmp.b.plot_annotation_multiline allows for plotting single-value annotations, where each label type is not only assigned to an individual plotting style, but also to a spatially separate row (or column).
fig, ax = libfmp.b.plot_annotation_multiline(ann, label_keys=label_keys,
sort_labels=['measure', 'quarter', 'eighth'])
ax.set_xlim([-1,3]);
The function libfmp.b.plot_annotation_multiline can be horizontally or vertically aligned to a given matrix, as demonstrated in the next code cell. In this example, we also show how two plots may be placed next to each other using the function libfmp.b.FloatingBox.
float_box = libfmp.b.FloatingBox()
fig, ax = plt.subplots(2, 1, sharex=True,
gridspec_kw={'height_ratios': [2, 1]}, figsize=(4, 3))
ax[0].imshow(matrix, cmap='gray_r', aspect='auto',
extent=[-1, 5, 0, 1], alpha=0.5)
ax[0].set_xlim([-1, 3])
libfmp.b.plot_annotation_multiline(ann, ax=ax[1], label_keys=label_keys,
sort_labels=['measure', 'quarter', 'eighth'],
time_min=-1, time_max=3)
plt.tight_layout()
float_box.add_fig(fig)
fig, ax = plt.subplots(1, 2, sharey=True, gridspec_kw={'width_ratios': [2, 10]},
figsize=(4, 3))
ax[1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[0, 1, -1, 3], alpha=0.5)
ax[1].set_ylim([-1, 5])
libfmp.b.plot_annotation_multiline(ann, ax=ax[0], label_keys=label_keys, sort_labels=['measure', 'quarter', 'eighth'],
direction='vertical', time_min=-1, time_max=3)
plt.tight_layout()
float_box.add_fig(fig)
float_box.show()
Segment Annotations¶
A second type of annotation we regularly encounter in the FMP notebooks are segment annotations, where each instance consists of two numerical values and a label. Typically, the two numerical values encode the start and end positions (given in seconds) of a time interval and the label assigns to this time interval a semantic category. For example, in chord recognition, the segments specify time intervals of a music recording and the labels specify musical chords such as $\mathrm{C}$-major or a $\mathrm{E}^\flat$-minor chord. Or, in music structure analysis, the segments encode musical parts, which are typically labeled by capital letters $A,B,C, \ldots$ in the order of their first occurrence. In the following code cell, we generate a toy example of such a segment annotation, which serves as our running example.
ann = [(0, 8, 'A'), (8, 16, 'A'), (16, 24, 'B'),
(27, 33, 'C'), (33, 40, 'C'), (40, 48, 'B')]
print('=== Dummy annotation ===')
print(ann)
# Convert to DataFrame and CSV format (as used for the FMP notebooks)
ann_df = pd.DataFrame(ann, columns=['start', 'end', 'label'])
ann_csv = ann_df.to_csv(sep=';', index=False, quoting=2)
print('\n=== Dummy annotation in CSV format ===')
print(ann_csv)
print('=== Dummy annotation stored as DataFrame and rendered as HTML ===')
ipd.display(ipd.HTML(ann_df.to_html()))
# Store and load annotation file
#fn_out = os.path.join('..', 'output', 'B', 'FMP_annotation_segment_dummy.csv')
#ann_df.to_csv(fn_out, sep=';', index=False, quoting=2)
#ann_df = pd.read_csv(fn_out, sep=';')
#ann = [(start, end, label) for i, (start, end, label) in df.iterrows()]
The function libfmp.b.plot_segments allows for plotting segment annotations, where each label type is assigned to an individual color. In the following code cell, we visualize the segment annotations with default colors and colors that are defined via a dictionary.
fig, ax = libfmp.b.plot_segments(ann)
color_ann = {'A': [1, 0, 0, 0.2], 'B': [0, 1, 0, 0.2],
'C': [0, 0, 1, 0.2], '': [1, 1, 1, 0.2]}
fig, ax = libfmp.b.plot_segments(ann, colors = color_ann)
Next, we visualize the segment annotations horizontally aligned to a matrix, once in vertical direction and once in horizontal direction.
float_box = libfmp.b.FloatingBox()
# Horizontal alignment
fig, ax = plt.subplots(2, 1, sharex=True,
gridspec_kw={'height_ratios': [4, 1]}, figsize=(3.5, 2.5))
ax[0].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-2, 50, 0, 1])
libfmp.b.plot_segments(ann, ax=ax[1], time_min=-2, time_max=50)
plt.tight_layout()
float_box.add_fig(fig)
# Vertical alignment
fig, ax = plt.subplots(1, 2, sharey=True, figsize=(3.5, 2.5),
gridspec_kw={'width_ratios': [1, 10]})
ax[1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[0, 1, -2, 50])
libfmp.b.plot_segments(ann, ax=ax[0], direction='vertical', time_min=-2, time_max=50)
plt.tight_layout()
float_box.add_fig(fig)
float_box.show()
One may also apply the alignment in both directions.
fig, ax = plt.subplots(2, 2, figsize=(5, 3), sharex='col', sharey='row',
gridspec_kw={'height_ratios': [6, 1], 'width_ratios': [1, 12]})
ax[0, 1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-2, 50, -2, 50])
libfmp.b.plot_segments(ann, ax=ax[1, 1], time_min=-2, time_max=50)
libfmp.b.plot_segments(ann, ax=ax[0, 0], direction='vertical', time_min=-2, time_max=50)
ax[1, 0].axis('off')
plt.tight_layout()
The function libfmp.b.plot_segments_overlay is provided for overlaying segment annotations with image plots. Among others, one can set a parameter (alpha) for controlling the transparency of the annotations' visualization and parameters (print_labels, label_ticks) for removing and placing labels. The next code cell shows some settings.
fig, ax = plt.subplots(1, 3, figsize=(8, 2.5))
ax[0].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-2, 50, 0, 1])
libfmp.b.plot_segments_overlay(ann, ax=ax[0], time_min=-2, time_max=50, alpha=0.8)
ax[1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-2, 50, 0, 1])
libfmp.b.plot_segments_overlay(ann, ax=ax[1], time_min=-2, time_max=50,
print_labels=False, alpha=0.3)
ax[2].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-2, 50, 0, 1])
libfmp.b.plot_segments_overlay(ann, ax=ax[2], time_min=-2, time_max=50,
label_ticks=True, alpha=0.5)
plt.tight_layout()
One may also overlay several horizontally and vertically aligned annotations. This is demonstrated by the next example.
ann_v = [(0.1, 0.2, 'X'), (0.2, 0.38, 'Y'), (0.4, 0.5, 'X'), (0.6, 0.7, 'X'), (0.7, 0.88, 'Y')]
color_ann_v = {'X': [1, 0, 0, 1], 'Y': [1, 1, 0, 1], '': [1, 1, 1, 1]}
fig, ax = plt.subplots(2, 2, figsize=(6, 4),
gridspec_kw={'height_ratios': [8, 1], 'width_ratios': [1, 12]})
ax[0,1].imshow(matrix, cmap='gray_r', aspect='auto', extent=[-10, 60, 0, 1])
libfmp.b.plot_segments_overlay(ann, ax=ax[0, 1], time_min=-10, time_max=60,
label_ticks=True, print_labels=False, alpha=0.6)
libfmp.b.plot_segments_overlay(ann_v, colors=color_ann_v, ax=ax[0, 1], time_min=0, time_max=1,
direction='vertical', label_ticks=True,
print_labels=False, alpha=0.4)
libfmp.b.plot_segments(ann, ax=ax[1, 1], time_min=-10, time_max=60)
libfmp.b.plot_segments(ann_v, ax=ax[0, 0], colors=color_ann_v,
direction='vertical', time_min=0, time_max=1)
ax[1,0].axis('off');
 |
 |
 |
 |
 |
 |
 |
 |
 |