Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Evaluating Impairment

Audio Assessment

• Objective Methods
  • Simple physical measures (SNR...)
  • Perceptually motivated (PEAQ, POLQA)
• Subjective Methods

Subjective Evaluation

1. Conducting listening experiments
2. Time-Consuming
3. Limited reproducibility in case no formal evaluation method was used

Evaluating Audio Codecs

Evaluating Audio Codecs

ITU Recommendation

• ITU-R BS.1534-0 published 2001
• 2nd revision just recently published
• introducing changes in the test design as well as statistical analysis of the results

MUltiple Stimuli

MUltiple Stimuli

Basic Audio Quality

“a single, global attribute to judge any and all detected differences between the reference and the object.”

MUltiple Stimuli

Hidden Reference

...and Anchor

Anchors?

• Ensure the the full range of the grading scale is used
• 3.5 kHz / 7.0 kHz low pass filtered signals
• Need to meet specific filter design criteria

Requirements

• Good performance (avoiding dropped samples)

Requirements

• Good performance (avoiding dropped samples)
• Sample accurate switching
• No crossfading!
• Instead: Fade-out and Fade-in of 5ms length

Switching

Not with <embed> or <audio>!

Software Design

• HTML5, JavaScript and Web Audio API
• All processing in ScriptProcessorNode
• UI in jQuery and jQuery Mobile
• Easy to extend, modular software design
• Easy to use for experimenters

Config in YAML Pages

pages:
  - type: mushra
    id: Item 1
    name: Orchestra
    content: Try to concentrate on the violins!
    showWaveform: true
    reference: reference.wav
    createAnchor35: true
    createAnchor70: true
    stimuli:
      C1: codec1.wav
      C2: codec2.wav
      C3: codec3.wav
					

Web Based Experiments

• Many types of auditory experiments can be done online
• Schoeffler/Stoeter 2013: By comparing the results of a laboratory experiment and a web experiment, only minor differences were found.

The future of auditory web experiments

• enabling large scale experiments
• using technologies like
  • WebSockets
  • WebRTC
• Experiment server (including live statistic)
• still room for improvements
• e.g. dropped samples, latency, channel mapping

Conclusion

• Web Audio API will bring new types of auditory experiments to the web
• Case Study: webMUSHRA - a (hopefully) recommendation-compliant MUSHRA software
  • Source code release in preparation

References / Links

• http://www.audiolabs-erlangen.de/resources/webMUSHRA