International Audio Laboratories Erlangen

Automated Exploration of Music Documents Exploiting Different Data Representations (ARMADA)

Logo_DFG Teaser_ARMADA Logo_UniBonn Logo_MPII Logo_UniSaarland

In the ARMADA project, we developed robust and efficient methods that allow users to access, explore, and analyze complex and inhomogeneous music collections. The project was funded by the German Research Foundation. On this website, we summarize the project's main outcomes and provide links to project-related resources (data, demonstrators, websites) and publications.

Project Description

Automated Exploration of Music Documents Exploiting Different Data Representations

Teaser_ARMADA_sync

In this project, we developed robust and efficient methods for the automated exploration of complex, inhomogeneous music collections. In addition to audio data (CD, MP3), these collections also contain text-based data (metadata, lyrics, libretti), symbolic score data (MusicXML, Capella, MIDI), and image data (scanned scores). Standard approaches to automated music data processing, which often rely on a single document of a certain data type, quickly reach their limits. In this research project, we pursued a novel approach to data processing in which we exploited the existence of different representation types of the same piece of music. On the one hand, we developed generic methods for automatically linking and synchronizing semantically related music representations of different types. We then used the calculated linking structures to support or even enable specific music information retrieval tasks such as automated annotation, structural analysis, and transcription of pieces of music. In order to evaluate and demonstrate the practical relevance of our methods, we integrated them into a software system (called SyncPlayer), which allows users to browse, analyze or simply enjoy music in various forms.

Projektbeschreibung

Automatisierte Erschließung von Musikdokumenten unter Ausnutzung verschiedener Darstellungsformen

Teaser_ARMADA_sync

In diesem Projekts ging es um die Entwicklung robuster und effizienter Verfahren zur automatisierten Erschließung komplexer inhomogener Musikdatenbestände, die neben Audiodaten (CD, MP3) auch textbasierte Daten (Metadaten, Liedtexte, Libretti), symbolische Partiturdaten (MusicXML, Capella, MIDI) oder Bilddaten (gescannte Partituren) enthalten. Standardansätze zur automatisiserten Musikdatenerschließung, die oft nur die Kenntnis eines einzelnen Dokuments eines bestimmten Datentyps voraussetzen, stoßen bei solchen Aufgaben schnell an ihre Grenzen. In diesem Forschungsvorhaben wurde ein neuartiger Ansatz zur Datenerschließung verfolgt werden, bei dem das Vorliegen unterschiedlicher Darstellungsformen ein und desselben Musikstücks systematisch ausgenutzt werden soll. Hierzu wurden einerseits generische Methoden zur automatisierten Verlinkung und Synchronisation semantisch in Beziehung stehender Musikdaten unterschiedlicher Formate entwickelt. Die berechneten Verlinkungs- und Synchronisationsstrukturen wurden dann verwendet werden, um konkrete Aufgaben des Music Information Retrieval wie die automatisierte Annotation, Strukturanalyse oder Transkription von Musikstücken zu unterstützen oder gar erst zu ermöglichen. Zur Evaluation und Demonstration der Praxisrelevanz der zu entwickelnden Methoden wurden diese in ein Software-System (SyncPlayer) integriert, das es dem Benutzer erlaubt, Musik in unterschiedlichen Erscheinungsformen zu durchsuchen, zu analysieren oder einfach nur zu genießen.

Projected-Related Resources and Demonstrators

The following list provides an overview of the most important publicly accessible sources created in the ARMADA project:

Projected-Related Publications

The following publications reflect the main scientific contributions of the work carried out in the ARMADA project.

  1. David Damm, Christian Fremerey, Verena Thomas, Michael Clausen, Frank Kurth, and Meinard Müller
    A digital library framework for heterogeneous music collections: From document acquisition to cross-modal interaction
    International Journal on Digital Libraries: Special Issue on Music Digital Libraries, 12(2-3): 53–71, 2012. PDF Details 
    @article{DammFTCKM12_DML_IJDL,
author    = {David Damm and Christian Fremerey and Verena Thomas and Michael Clausen and Frank Kurth and Meinard M{\"u}ller},
title     = {A digital library framework for heterogeneous music collections: {F}rom document acquisition to cross-modal interaction},
journal   = {International Journal on Digital Libraries: Special Issue on Music Digital Libraries},
volume    = {12},
number    = {2-3},
year      = {2012},
pages     = {53--71},
url-details = {https://link.springer.com/article/10.1007/s00799-012-0087-y},
url-pdf   = {2012_DammFTCKM_DML_IJDL.pdf}
}
  2. Sebastian Ewert and Meinard Müller
    Refinement Strategies for Music Synchronization
    In Proceedings of the International Symposium on Computer Music Modeling and Retrieval (CMMR): 147–165, 2008. PDF Details 
    @inproceedings{EwertM08_RefinementStrategies_CMMR,
author    = {Sebastian Ewert and Meinard M{\"u}ller},
title     = {Refinement Strategies for Music Synchronization},
booktitle = {Proceedings of the International Symposium on Computer Music Modeling and Retrieval ({CMMR})},
address   = {Copenhagen, Denmark},
series    = {Lecture Notes in Computer Science},
volume    = {5493},
isbn      = {978-3-642-02517-4},
month     = may,
year      = {2008},
pages     = {147--165},
url-pdf   = {2008_EwertMueller_RefinementStrategies_CMMR.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/SyncRWC60}
}
  3. Sebastian Ewert and Meinard Müller
    Score-Informed Voice Separation For Piano Recordings
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 245–250, 2011. PDF 
    @inproceedings{EwertMueller11_VoiceSeparation_ISMIR,
author    = {Sebastian Ewert and Meinard M{\"u}ller},
title     = {Score-Informed Voice Separation For Piano Recordings},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
address   = {Miami, Florida, USA},
year      = {2011},
pages     = {245--250},
url-pdf   = {2011_EwertMueller_VoiceSeparationPiano_ISMIR.pdf},
}
  4. Sebastian Ewert and Meinard Müller
    Estimating Note Intensities in Music Recordings
    In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 385–388, 2011. PDF DOI 
    @inproceedings{EwertM11_NoteIntensities_ICASSP,
author     = {Sebastian Ewert and Meinard M{\"u}ller},
title      = {Estimating Note Intensities in Music Recordings},
booktitle  = {Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
address  = {Prague, Czech},
year       = {2011},
pages      = {385--388},
doi        = {10.1109/ICASSP.2011.5946421},
url-pdf   = {2011_EwertMueller_DynamicsEstimation_ICASSP.pdf}
}
  5. Sebastian Ewert and Meinard Müller
    Using Score-Informed Constraints for NMF-based Source Separation
    In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 129–132, 2012. PDF Demo 
    @inproceedings{EwertM12_ScoreInformedNMF_ICASSP,
author     = {Sebastian Ewert and Meinard M{\"u}ller},
title      = {Using Score-Informed Constraints for {NMF}-based Source Separation},
booktitle  = {Proceedings of the {IEEE} International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
address  = {Kyoto, Japan},
year       = {2012},
pages    = {129--132},
url-pdf   = {2012_EwertMueller_ScoreConstrainedNMF_ICASSP.pdf},
url-demo = {http://resources.mpi-inf.mpg.de/MIR/ICASSP2012-ScoreInformedNMF/}
}
  6. Sebastian Ewert, Meinard Müller, Verena Konz, Daniel Müllensiefen, and Gerraint A. Wiggins
    Towards Cross-Version Harmonic Analysis of Music
    IEEE Transactions on Multimedia, 14(3-2): 770–782, 2012. Details DOI 
    @article{EwertMKMW12_CrossDomainHarmonicAnalysis_IEEE-TMM,
author    = {Sebastian Ewert and Meinard M{\"u}ller and Verena Konz and Daniel M{\"u}llensiefen and Gerraint A. Wiggins},
title     = {Towards Cross-Version Harmonic Analysis of Music},
journal   = {IEEE Transactions on Multimedia},
volume    = {14},
number    = {3-2},
year      = {2012},
pages     = {770--782},
url-details = {https://ieeexplore.ieee.org/document/6165370},
doi       = {10.1109/TMM.2012.2190047}
}
  7. Sebastian Ewert, Meinard Müller, and Michael Clausen
    Towards timbre-invariant audio features for harmony-based music
    In Proceedings of the International Conference on Acoustics (NAG/DAGA): 352–353, 2009. PDF 
    @inproceedings{EwertMC09_TimbreInvariantAudioFeatures_DAGA,
author    = {Sebastian Ewert and Meinard M{\"u}ller and Michael Clausen},
title     = {Towards timbre-invariant audio features for harmony-based music},
booktitle = {Proceedings of the International Conference on Acoustics ({NAG/DAGA})},
year      = {2009},
address   = {Rotterdam, Netherlands},
pages     = {352--353},
url-pdf   = {2009_EwertMuellerClausen_TimbreInvariantFeatures_DAGA.pdf},
}
  8. Sebastian Ewert, Meinard Müller, and Roger B. Dannenberg
    Towards Reliable Partial Music Alignments Using Multiple Synchronization Strategies
    In Proceedings of the International Workshop on Adaptive Multimedia Retrieval (AMR), 2009. PDF 
    @inproceedings{EwertMD09_ReliableAlignments_AMR,
author      = {Sebastian Ewert and  Meinard M{\"u}ller and Roger B. Dannenberg},
title       = {Towards Reliable Partial Music Alignments Using Multiple Synchronization Strategies},
booktitle   = {Proceedings of the International Workshop on Adaptive Multimedia Retrieval (AMR)},
address     = {Madrid, Spain},
year        = {2009},
month       = sep,
pages       = {},
url-pdf   = {2009_EwertMuellerDannenberg_ReliableAlignments_AMR.pdf},
}
  9. Sebastian Ewert, Meinard Müller, and Peter Grosche
    High Resolution Audio Synchronization Using Chroma Onset Features
    In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 1869–1872, 2009. PDF Details 
    @inproceedings{EwertMG09_HighResAudioSync_ICASSP,
author      = {Sebastian Ewert and Meinard M{\"u}ller and Peter Grosche},
title       = {High Resolution Audio Synchronization Using Chroma Onset Features},
booktitle   = {Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
address     = {Taipei, Taiwan},
month       = apr,
year        = {2009},
pages       = {1869--1872},
isbn        = {978-1-4244-2354-5},
issn        = {1520-6149},
url-pdf   = {2009_EwertMuellerGrosche_HighResAudioSync_ICASSP.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/SyncRWC60}
}
  10. Meinard Müller, Michael Clausen, Verena Konz, Sebastian Ewert, and Christian Fremerey
    A Multimodal Way of Experiencing and Exploring Music
    Interdisciplinary Science Reviews (ISR), 35(2): 138–153, 2010. PDF 
    @article{MuellerCKEF10_Sync_ISR,
author    = {Meinard M{\"u}ller and Michael Clausen and Verena Konz and Sebastian Ewert and Christian Fremerey},
journal   = {Interdisciplinary Science Reviews (ISR)},
title     = {A Multimodal Way of Experiencing and Exploring Music},
number    = {2},
publisher = {Maney},
volume    = {35},
year      = {2010},
pages     = {138--153},
url-pdf   = {2010_MuellerClausenKonzEwertFremerey_MusicSynchronization_ISR.pdf}
}
  11. Meinard Müller and Sebastian Ewert
    Joint Structure Analysis with Applications to Music Annotation and Synchronization
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 389–394, 2008. PDF 
    @inproceedings{MuellerE08_JointStructureAnalysis_ISMIR,
author    = {Meinard M{\"u}ller and Sebastian Ewert},
title     = {Joint Structure Analysis with Applications to Music Annotation and Synchronization},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
address   = {Philadelphia, Pennsylvania, USA},
isbn      = {978-0-615-24849-3},
month     = sep,
year      = {2008},
pages     = {389--394},
url-pdf   = {2008_MuellerEwert_JointStructureAnalysis_ISMIR.pdf},
}
  12. Meinard Müller and Sebastian Ewert
    Chroma Toolbox: MATLAB implementations for extracting variants of chroma-based audio features
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 215–220, 2011. PDF Demo 
    @inproceedings{MuellerEwert11_ChromaToolbox_ISMIR,
author    = {Meinard M{\"u}ller and Sebastian Ewert},
title     = {{C}hroma {T}oolbox: {MATLAB} implementations for extracting variants of chroma-based audio features},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
address   = {Miami, Florida, USA},
year      = {2011},
pages     = {215--220},
url-pdf   = {2011_MuellerEwert_ChromaToolbox_ISMIR.pdf},
url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/chromatoolbox}
}
  13. Meinard Müller and Sebastian Ewert
    Towards Timbre-Invariant Audio Features for Harmony-Based Music
    IEEE Transactions on Audio, Speech, and Language Processing, 18(3): 649–662, 2010. Demo DOI 
    @article{MuellerEwert10_CRP_TASLP,
author    = {Meinard M{\"u}ller and Sebastian Ewert},
journal   = {{IEEE} Transactions on Audio, Speech, and Language Processing},
title     = {Towards Timbre-Invariant Audio Features for Harmony-Based Music},
number    = {3},
publisher = {IEEE},
volume    = {18},
year      = {2010},
pages     = {649--662},
doi      = {10.1109/TASL.2010.2041394},
url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/chromatoolbox}
}
  14. Meinard Müller, Sebastian Ewert, and Sebastian Kreuzer
    Making Chroma Features More Robust to Timbre Changes
    In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 1869–1872, 2009. PDF Details 
    @inproceedings{MuellerEK09_ChromaFeaturesRobust_ICASSP,
author      = {Meinard M{\"u}ller and Sebastian Ewert and Sebastian Kreuzer},
title       = {Making Chroma Features More Robust to Timbre Changes},
booktitle   = {Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
address     = {Taipei, Taiwan},
month       = apr,
year        = {2009},
pages       = {1869--1872},
isbn        = {978-1-4244-2354-5},
issn        = {1520-6149},
url-pdf   = {2009_MuellerEwertKreuzer_ChromaFeaturesRobust_ICASSP.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/chromatoolbox}
}
  15. Meinard Müller, Verena Konz, Andi Scharfstein, Sebastian Ewert, and Michael Clausen
    Towards Automated Extraction of Tempo Parameters from Expressive Music Recordings
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 69–74, 2009. PDF 
    @inproceedings{MuellerKSEC09_TempoParametersFromRecordings_ISMIR,
author    = {Meinard M{\"u}ller and Verena Konz and Andi Scharfstein and Sebastian Ewert and Michael Clausen},
title     = {Towards Automated Extraction of Tempo Parameters from Expressive Music Recordings},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
year      = {2009},
month     = oct,
address   = {Kobe, Japan},
pages     = {69--74},
url-pdf   = {2009_MuellerKonzScharfsteinEwertClausen_TempoCurves_ISMIR.pdf},
}
  16. Verena Thomas, Christian Fremerey, Meinard Müller, and Michael Clausen
    Linking Sheet Music and Audio — Challenges and New Approaches
    In Meinard Müller and Masataka Goto and Markus Schedl (ed.): Multimodal Music Processing, Schloss Dagstuhl—Leibniz-Zentrum für Informatik, 3: 1–22, 2012. PDF 
    @InCollection{ThomasFMC12_LinkingSheetMusicAudio_DagstuhlFU,
author  = {Verena Thomas and Christian Fremerey and Meinard M{\"u}ller and Michael Clausen},
title   = {Linking Sheet Music and Audio -- Challenges and New Approaches},
booktitle = {Multimodal Music Processing},
series  = {Dagstuhl Follow-Ups},
pages   = {1--22},
year        = {2012},
volume  = {3},
editor  = {Meinard M{\"u}ller and Masataka Goto and Markus Schedl},
publisher = {Schloss Dagstuhl--Leibniz-Zentrum f\"ur Informatik},
address     = {Dagstuhl, Germany},
url-pdf   = {2012_ThomasFremereyMuellerClausen_LinkingSheetMusicAudio_DagstuhlFU.pdf}
}
  17. Verena Thomas, Christian Wagner, and Michael Clausen
    OCR based post processing of OMR for the recovery of transposing instruments in complex orchestral scores
    In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 411–416, 2011. PDF 
    @inproceedings{ThomasWC11_OCR-OMR_ISMIR,
author       = {Verena Thomas and Christian Wagner and Michael Clausen},
title        = {{OCR} based post processing of {OMR} for the recovery of transposing
instruments in complex orchestral scores},
booktitle    = {Proceedings of the International Society for Music Information Retrieval Conference  ({ISMIR})},
pages        = {411--416},
publisher    = {University of Miami},
year         = {2011},
url-pdf   = {2011_ThomasWC_OCR-OMR_ISMIR.pdf}
}

Projected-Related Ph.D. Theses

  1. Sebastian Ewert
    Signal Processing Methods for Music Synchronization, Audio Matching, and Source Separation
    PhD Thesis, Rheinischen Friedrich-Wilhelms-Universität Bonn, 2012. PDF Details 
    @phdthesis{Ewert12_SignalProcessingMusic_PhD,
author      = {Sebastian Ewert},
year        = {2012},
title       = {Signal Processing Methods for Music Synchronization, Audio Matching, and Source Separation},
school      = {Rheinischen Friedrich-Wilhelms-Universit{\"a}t Bonn},
url-details = {https://bonndoc.ulb.uni-bonn.de/xmlui/handle/20.500.11811/5410},
url-pdf = {2012_EwertSebastian_SignalProcessingMethods_Phd-Thesis.pdf}
}
  2. Verena Kriesel
    Music Synchronization, Audio Matching, Pattern Detection, and User Interfaces for a Digital Music Library System
    PhD Thesis, Rheinischen Friedrich-Wilhelms-Universität Bonn, 2012. PDF Details 
    @phdthesis{Kriesel13_MusicSyncMatchDetect_PhD,
author      = {Verena Kriesel},
year        = {2012},
title       = {Music Synchronization, Audio Matching, Pattern Detection, and User Interfaces for a Digital Music Library System},
school      = {Rheinischen Friedrich-Wilhelms-Universit{\"a}t Bonn},
url-details = {https://bonndoc.ulb.uni-bonn.de/xmlui/handle/20.500.11811/5714},
url-pdf = {2013_Kriesel_MusicSyncMatchDetect_PhD-Thesis.pdf}
}