Piano Concerto Mixing: A Computational Pipeline

In this project, we propose a computational pipeline that allows pianists of all skill levels to create their unique mixes with an orchestra track extracted from an existing recording. In particular, this pipeline consists of different components using techniques from Music Information Retrieval (MIR). First, starting with a open-source complete piano concerto recording, we employ data-driven MSS approaches to separate the piano and orchestra. Then, we use music synchronization techniques to temporally align the separated orchestra track with the pianist’s own recording. Finally, we apply time-scale modification (TSM) to warp the orchestra track and create the final mix.

IMSLP & YouTube

Chopin Piano Concerto No.2 in F Minor, Op.21, 1st Movement

Pianist: Tanya Bannister

Conductor: Mixhael Luxner

Orchestra: Millikin-Decatur Symphony Orchestra

Copyright: Public Domain

Source: International Music Score Library Project (IMSLP)

Separation, Measures 86–90

Orchestra + Piano (OP)

Piano

Orchestra

Synchronization + TSM

Measures 86–90

Full recording

Chopin Piano Concerto No.2 in F Minor, Op.21, 1st Movement

Pianist: Arthur Rubinstein

Conductor: Stanisław Skrowaczewski

Orchestra: New Symphony Orchestra of London

Year: 1962

Copyright: Public Domain

Source: YouTube

Separation, Measures 86 – 90

Orchestra + Piano (OP)

Piano

Orchestra

Beethoven Piano Concerto No.1 in C Major, Op.21, 1st Movement

Pianist: Arthur Schnabel

Conductor: Malcolm Sargent

Orchestra: London Philarmonic Orchestra

Year: 1932

Copyright: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 - Non-PD US

Source: International Music Score Library Project (IMSLP)

Separation, Measures 134–142

Orchestra + Piano (OP)

Piano

Orchestra

Synchronization + TSM, Measures 134–142

Rachmaninov Piano Concerto No.2 in C Minor, Op.18, 1st Movement

Pianist: N/A

Conductor: Anthony Holland

Orchestra: Skidmore College Orchestra

Year: N/A

Copyright: Creative Commons Attribution Share Alike 3.0

Source: International Music Score Library Project (IMSLP)

Separation, Measures 245–252

Orchestra + Piano (OP)

Piano

Orchestra

Synchronization + TSM, Measures 245–252

PCD

J. S. Bach Piano Concerto in F Minor, BWV 1056, 1st Movement

Excerpt ID: Bach_BWV1056-01-mm001-008_YO-V2

Separation, Measures 1–8

Orchestra + Piano (OP)

Piano

Orchestra

Chopin Piano Concerto No.2 in F Minor, Op.23, 3rd Movement

Excerpt ID: Chopin_Op021-03-mm215-229_ES

Separation, Measures 215–229

Orchestra + Piano (OP)

Piano

Orchestra

Tchaikovsky Piano Concerto No.1 in B Flat Minor, Op.23, 1st Movement

Excerpt ID: Tschaikovsky_Op023-01-mm030-036_ES-V2

Separation, Measures 30–36

Orchestra + Piano (OP)

Piano

Orchestra

References

1. Yigitcan Özer and Meinard Müller
   Source Separation of Piano Concertos Using Musically Motivated Augmentation Techniques
   Submitted to IEEE/ACM Transactions on Audio, Speech & Language Processing, 2023. Bibtex Demo

   @article{OezerM23_PCSeparation_TASLP,
   title     = {Source Separation of Piano Concertos Using Musically Motivated Augmentation Techniques},
   author    = {Yigitcan {\"O}zer and Meinard M{\"u}ller},
   journal   = {Submitted to {IEEE/ACM} Transactions on Audio, Speech {\&} Language Processing},
   volume    = {},
   number    = {},
   year      = {2023},
   pages     = {},
   url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/2023-PianoConcertoSeparation}
   }

2. Yigitcan Özer and Meinard Müller
   Source Separation of Piano Concertos with Test-Time Adaptation
   In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 493–500, 2022. Bibtex Demo

   @inproceedings{OezerM22_PianoSepAdapt_ISMIR,
   author    = {Yigitcan \"Ozer and Meinard M\"uller},
   title     = {Source Separation of Piano Concertos with Test-Time Adaptation},
   booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
   address   = {Bengaluru, India},
   year      = {2022},
   pages     = {493--500},
   url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/2022-PianoSep}
   }

3. Alexandre Défossez
   Hybrid Spectrogram and Waveform Source Separation
   In Proceedings of the ISMIR 2021 Workshop on Music Source Separation, 2021. Bibtex

   @inproceedings{Defossez21_Demucs_ISMIR,
   author    = {Alexandre Défossez},
   title     = {Hybrid Spectrogram and Waveform Source Separation},
   booktitle = {Proceedings of the {ISMIR} 2021 Workshop on Music Source Separation},
   year      = {2021},
   address   = {Online}
   }

4. Meinard Müller, Yigitcan Özer, Michael Krause, Thomas Prätzlich, and Jonathan Driedger
   Sync Toolbox: A Python Package for Efficient, Robust, and Accurate Music Synchronization
   Journal of Open Source Software (JOSS), 6(64): 1–4, 2021. Bibtex PDF Demo DOI

   @article{MuellerZ21_SyncToolbox_JOSS,
   author    = {Meinard M{\"u}ller and Yigitcan {\"O}zer and Michael Krause and Thomas Pr{\"a}tzlich and Jonathan Driedger},
   title     = {{S}ync {T}oolbox: {A} {P}ython Package for Efficient, Robust, and Accurate Music Synchronization},
   journal   = {Journal of Open Source Software ({JOSS})},
   volume    = {6},
   number    = {64},
   year      = {2021},
   pages     = {3434:1--4},
   doi         = {10.21105/joss.03434},
   url-pdf   = {2021_MuellerOKPD_SyncToolbox_JOSS.pdf},
   url-demo = {https://github.com/meinardmueller/synctoolbox}
   }

5. Thomas Prätzlich, Jonathan Driedger, and Meinard Müller
   Memory-Restricted Multiscale Dynamic Time Warping
   In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 569–573, 2016. Bibtex PDF

   @inproceedings{PraetzlichDM16_MsDTW_ICASSP,
   author    = {Thomas Pr{\"a}tzlich and Jonathan Driedger and Meinard M{\"u}ller},
   title     = {Memory-Restricted Multiscale Dynamic Time Warping},
   booktitle = {Proceedings of the {IEEE} International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})},
   address   = {Shanghai, China},
   year      = {2016},
   pages     = {569--573},
   url-pdf   = {2016_PraetzlichDriedgerMueller_MrMsDTW_ICASSP.pdf}
   }

6. Jonathan Driedger, Meinard Müller, and Sebastian Ewert
   Improving Time-Scale Modification of Music Signals using Harmonic—Percussive Separation
   IEEE Signal Processing Letters, 21(1): 105–109, 2014. Bibtex Demo

   @article{DriedgerME14_HPTSM_IEEE-SPL,
   author    = {Jonathan Driedger and Meinard M{\"u}ller and Sebastian Ewert},
   title     = {Improving Time-Scale Modification of Music Signals using Harmonic--Percussive Separation},
   journal   = {{IEEE} Signal Processing Letters},
   volume    = {21},
   number    = {1},
   year      = {2014},
   pages     = {105--109},
   url-demo = {https://www.audiolabs-erlangen.de/resources/2014-SPL-HPTSM/}
   }

7. Yigitcan Özer, Simon Schwär, Vlora Arifi-Müller, Jeremy Lawrence, Emre Sen, and Meinard Müller
   Piano Concerto Dataset (PCD): A Multitrack Dataset of Piano Concertos
   Transactions of the International Society for Music Information Retrieval (TISMIR), 6(1): 75–88, 2023. Bibtex Demo DOI

   @article{OezerSALSM23_PCD_TISMIR,
   title     = {{P}iano {C}oncerto {D}ataset {(PCD)}: A Multitrack Dataset of Piano Concertos},
   author    = {Yigitcan \"Ozer and Simon Schw\"ar and Vlora Arifi-M\"uller and Jeremy Lawrence and Emre Sen and Meinard M\"uller},
   journal   = {Transactions of the International Society for Music Information Retrieval ({TISMIR})},
   volume    = {6},
   number    = {1},
   year      = {2023},
   pages     = {75--88},
   doi       = {10.5334/tismir.160},
   url-demo  = {https://www.audiolabs-erlangen.de/resources/MIR/PCD}
   }