webMUSHRA — A Comprehensive Framework for Web-based Listening Tests

Michael Schoeffler; Sarah Bartoschek; Fabian-Robert Stöter; Marlene Roess; Susanne Westphal; Bernd Edler; Jürgen Herre

doi:10.5334/jors.187

Abstract

For a long time, many popular listening test methods, such as ITU-R BS.1534 (MUSHRA), could not be carried out as web-based listening tests, since established web standards did not support all required audio processing features. With the standardization of the Web Audio API, the required features became available and, therefore, also the possibility to implement a wide range of established methods as web-based listening tests. In order to simplify the implementation of MUSHRA listening tests, the development of webMUSHRA was started. By utilizing webMUSHRA, experimenters can configure web-based MUSHRA listening tests without the need of web programming expertise. Today, webMUSHRA supports many more listening test methods, such as ITU-R BS.1116 and forced-choice procedures. Moreover, webMUSHRA is highly customizable and has been used in many auditory studies for different purposes.

References

1Punch, J L, Rakerd, B and Amlani, A M 2001 Paired-comparison hearing aid preferences: Evaluation of an unforced-choice paradigm. Journal Americal Academy of Audiology, 12: 190–201.
Back to article
2International Telecommunication Union 2015 Recommendation ITU-R BS.1534-3: Method for the subjective assessment of intermediate quality level of audio systems. https://www.itu.int/rec/R-REC-BS.1534/en.
Back to article
3International Telecommunication Union 2015 Recommendation ITU-R BS.1116-3: Methods for the subjective assessment of small impairments in audio systems. https://www.itu.int/rec/R-REC-BS.1116/en.
Back to article
4Schoeffler, M, Stöter, F-R, Bayerlein, H, Edler, B and Herre, J 2013 An Experiment about Estimating the Number of Instruments in Polyphonic Music: A Comparison Between Internet and Laboratory Results. Proceedings of 14th International Society for Music Information Retrieval Conference. Curitiba, Brazil, International Society for Music Information Retrieval. http://www.ppgia.pucpr.br/ismir2013/wp-content/uploads/2013/09/59_Paper.pdf.
Back to article
5Cartwright, M, Pardo, B, Mysore, G J and Hoffman, M 2016 Fast and easy crowdsourced perceptual audio evaluation. Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, 619–623. IEEE.
Back to article
6Schoeffler, M, Stöter, F-R, Edler, B and Herre, J 2015 Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA). 1st Web Audio Conference. Paris, France. http://wac.ircam.fr/pdf/wac15_submission_8.pdf.
Back to article
7Kraft, S and Zölzer, U 2014 BeaqleJS: HTML5 and JavaScript based Framework for the Subjective Evaluation of Audio Quality. Linux Audio Conference. Karlsruhe, Germany.
Back to article
8Jillings, N, Moffat, D, De Man, B and Reiss, J D 2015 Web Audio Evaluation Tool: A browser-based listening test environment. 12th Sound and Music Computing Conference. July.
Back to article
9Novo, P 2005 Auditory virtual environments. Communication Acoustics, 277–297. Springer Berlin Heidelberg.
Back to article
10International Telecommunication Union 2003 Recommendation ITU-R BS.1284-1: General methods for the subjective assessment of sound quality. https://www.itu.int/rec/R-REC-BS.1284/en.
Back to article
11Zielinski, S K, Rumsey, F and Bech, S 2008 On some biases encountered in modern audio quality listening tests-a review. J. Audio Eng. Soc., 56: 427–451. http://www.aes.org/e-lib/browse.cfm?elib=14393.
Back to article
12Liebetrau, J, Nagel, F, Zacharov, N, Watanabe, K, Colomes, C, Crum, P, Sporer, T and Mason, A 2014 Revision of Rec. ITU-R BS.1534. Audio Engineering Society Convention 137, 1–8. Los Angeles, United States, Oct. http://www.aes.org/e-lib/browse.cfm?elib=17495.
Back to article
13Schoeffler, M 2017 Overall Listening Experience — a new Approach to Subjective Evaluation of Audio. Ph.D. thesis. https://opus4.kobv.de/opus4-fau/files/8290/dissertation_schoeffler.pdf.
Back to article
14Schoeffler, M, Conrad, S and Herre, J 2014 The Influence of the Single/Multi-Channel-System on the Overall Listening Experience. Proc. of the AES 55th Conference on Spatial Audio. Helsinki, Finland. http://www.aes.org/e-lib/browse.cfm?elib=17350.
Back to article
15Schoeffler, M, Gernert, J L, Neumayer, M, Westphal, S and Herre, J 2015 On the validity of virtual reality-based auditory experiments: A case study about ratings of the overall listening experience. Springer Virtual Reality, 1–20. http://www.mschoeffler.de/wp-content/uploads/2017/01/manuscript_springer_vr__Validity_Virtual_Reality_Auditory_Experiments.pdf.
Back to article
16Schoeffler, M, Silzle, A and Herre, J 2017 Evaluation of spatial/3d audio: Basic audio quality versus quality of experience. IEEE Journal of Selected Topics in Signal Processing, 11: 75–88. http://ieeexplore.ieee.org/document/7782315/. DOI: 10.1109/JSTSP.2016.2639325
Back to article
17Westphal, S, Schoeffler, M and Herre, J 2015 A framework for reporting spatial attributes of sound sources. Workshop für Innovative Computerbasierte Musikinterfaces (ICMI). http://www.icmi-workshop.org/papers/2015/spacialattributes.pdf.
Back to article
18Morimoto, M 1997 The Role of Rear Loudspeakers in Spatial Impression. Audio Engineering Society Convention 103. New York, United States. http://www.aes.org/e-lib/browse.cfm?elib=7225.
Back to article
19Norcross, S G, Soulodre, G A and Lavoie, M C 2003 Temporal Aspects of Listener Envelopment in Multichannel Surround Systems. Audio Engineering Society Convention 114. Amsterdam, Netherlands.
Back to article
20Paine, G, Sazdov, R and Stevens, K 2007 Perceptual investigation into envelopement, spatial clarity, and engulfment in reproduced multi-channel audio. Audio Engineering Society Conference: 31st International Conference: New Directions in High Resolution Audio. Jun. http://www.aes.org/e-lib/browse.cfm?elib=13961.
Back to article
21Haber, L, Haber, R N, Penningroth, S, Novak, K and Radgowski, H 1993 Comparison of nine methods of indicating the direction to objects: data from blind adults. Perception, 22: 35–47. http://www.ncbi.nlm.nih.gov/pubmed/8474833. DOI: 10.1068/p220035
Back to article
22Schoeffler, M, Westphal, S, Adami, A, Bayerlein, H and Herre, J 2014 Comparison of a 2D- and 3D-Based Graphical User Interface for Localization Listening Tests. Proc. of the EAA Joint Symposium on Auralization and Ambisonics, 107–112. Berlin, Germany. https://depositonce.tu-berlin.de/handle/11303/175.
Back to article
23Stöter, F-R, Schoeffler, M, Edler, B and Herre, J 2013 Human ability of counting the number of instruments in polyphonic music. Proceedings of Meetings on Acoustics, 19. Montreal, Canada, Acoustical Society of America. http://asa.scitation.org/doi/10.1121/1.4805760.
Back to article
24Adami, A, Schoeffler, M and Herre, J 2015 Re-Panning of Directional Signals and its Impact on Localization. 23rd European Signal Processing Conference (EUSIPCO), 549–553. http://ieeexplore.ieee.org/document/7362443/.
Back to article
25Adami, A, Disch, S, Steba, G and Herre, J 2016 Assessing Applause Density Perception Using Synthesized Layered Applause Signals. Proceedings of the 19th International Conference on Digital Audio Effects (DAFx-16), 183–189. Brno, Czech Republic. http://dafx16.vutbr.cz/dafxpapers/26-DAFx-16_paper_15-PN.pdf.
Back to article
26Neumayer, M, Schoeffler, M and Herre, J 2015 Distributed pair evaluation in context of mushra listening tests. Workshop für Innovative Computerbasierte Musikinterfaces (ICMI). http://www.icmi-workshop.org/papers/2015/mushra.pdf.
Back to article

webMUSHRA — A Comprehensive Framework for Web-based Listening Tests

Abstract

Paradigm

My account