Conference PaperPDF Available

Emerging W3C APIs opened up commercial opportunities for computer music applications

Authors:

Abstract and Figures

In 2018, with a group of researchers and developers (some of whom are members of the W3C WebAudio WG) we proposed a WebAudio Plugin standard (WAP) and gave birth to a growing ecosystem for the development of computer music applications in the browser. These plugins can be seen as a transposition of what exists in the native world, adapted to be web-aware (i.e. a plugin can be remotely included in any host web audio application by a URI). Since then, many contributions have been made and many plugins have been developed. Not only several "host" applications have appeared, including commercial ones, but also tools to help developers, including an online IDE coded as a Progressive Web App (PWA) to write and publish plugins from A to Z, the audio DSP core of the plugins being compiled into WebAssembly. Some of the plugins we developed during the French research project WASABI are now sold as add-ons to an online Digital Audio Workstation (Amped Studio), showing the new opportunities that these emerging APIs have created.
Content may be subject to copyright.
Emerging W3C APIs opened up commercial
opportunities for computer music applications
Michel Buffa, Jerome Lebrun
Université Côte d’Azur
CNRS, INRIA
(buffa, lebrun)@i3s.unice.fr
Shihong Ren, Stéphane Letz, Yann
Orlarey, Romain Michon, Dominique Fober
GRAME, 11 cours de Verdun LYON
renshihong@hotmail.com
(letz, orlarey, michon, fober)@grame.fr
Introduction
In 2018, with a group of researchers and developers (some of whom are members of the W3C WebAudio
WG) we proposed a WebAudio Plugin standard (WAP) and gave birth to a growing ecosystem for the
development of computer music applications in the browser [1]. These plugins can be seen as a transposition
of what exists in the native world, adapted to be web-aware (i.e. a plugin can be remotely included in any host
web audio application by a URI). Since then, many contributions have been made and many plugins have been
developed. Not only several "host" applications have appeared, including commercial ones, but also tools to
help developers, including an online IDE coded as a Progressive Web App (PWA) to write and publish
plugins from A to Z, the audio DSP core of the plugins being compiled into WebAssembly. Some of the
plugins we developed during the French research project WASABI are now sold as add-ons to an online
Digital Audio Workstation (Amped Studio), showing the new opportunities that these emerging APIs have
created.
The emergence of WebAssembly for DSP algorithms
Plugins can be written 1) in pure JavaScript, only using the high-level features of the WebAudio API, or 2)
can be written in C/C++ or using DSLs and cross-compiled to WebAssembly. Rapidly, it has become
important to provide tools to facilitate the development of DSPs that can be compiled to this new standard.
Many developers used to code native plugins in C or C++ have gotten into the habit of prototyping the DSP
part with DSLs like FAUST [6]. This is why we recently designed a new IDE for this language, which takes
advantage not only of WebAssembly (the FAUST compiler itself was compiled in WebAssembly [7]), but
also of other standards emerging from the W3C (PWA, WebMIDI, WebAudio, WebComponents, etc.).
An online IDE for RAD development of WebAssembly-based WebAudio
Plugins
This online IDE facilitates the development of audio plugins to be deployed on different targets (different
native plugin formats are supported) but we will focus here on the production of WebAudio plugins [2]. The
IDE embeds a multi-file source code editor that supports IntelliSense-like features such as syntax highlighting
and auto-completion. It also includes the WebAssembly version of the FAUST compiler, enabling developers
to write, compile, run and test their code directly in the browser thanks to the WebAudio API. This IDE
supports multiple audio inputs (physical audio devices, MIDI interfaces, computer keyboard), different types
of signal visualizations (spectrogram, oscilloscope, etc.), and generates a default GUI for testing the DSP
coded (the FAUST language supports a set of primitives for describing abstract GUIs), as shown in Fig 1.
Fig 1: the online FAUST IDE showing a default GUI for testing the compiled code.
Fig 2. A GUI builder is embedded in the IDE. Plugins can be tested online with their custom built GUI,
downloaded as local zip files, published on remote WebAudio Plugins repositories.
PWA, GUI Builder and publishing to online remote plugin servers
Finally, the IDE embeds a GUI builder based on WebComponents (Fig 2.). One can develop an audio plugin
from scratch (or reuse code of existing ones), including a customized GUI, test it, download it as an archive (a
zip file) or deploy it to a remote WAP server so that host applications can “discover” the published plugins
automatically (Fig 4.). Plugins are all running WebAssembly code and the final GUI is based on
WebComponents. The IDE is a PWA and can be used offline with some limitations (i.e no remote publishing
of plugins).
Plugins and host applications developed so far...
More than 70 plugins have been developed so far by multiple contributors. A subset can be seen in Figures 3
and 4.
Guitar tuner, equalizers, mixers,
Tube guitar amplifier simulations [4, 5],
Many audio effects : delay, flanger, chorus, phasers, wah wah, stereo enhancers, several types of non
linear distortions (recreations of famous FX pedals for guitarists such as the Big Muff by
Electro-Harmonix, the Roland TS9 overdrive, ...), and many more to come.
Several instruments: analog synthesizers [8] (recreations of the Oberheim OBxD, Yamaha DX7, Korg
Minilogue, etc.), instruments based on physical modeling (djembé, guitar)...
Host web applications include:
The Virtual Pedalboard app for assembling plugins in a graph of audio effects and instruments [3],
shown in Fig 3, is a virtual recreation of the pedalboards that guitarists use on stage. These are
composed of a set of audio FX pedals connected together, the output of the pedalboard going to a
guitar amplifier.
A meta plugin called a “Rack” that is “a plugin for assembling plugins” that acts as a host and comes
with multiple banks and presets to produce famous guitar sounds, inspired by the “guitar pedal and
amp” manager of the GarageBand DAW application available on Apple devices (Fig 4.)
Digital Audio Workstations (DAWs, see Fig 5.) for recording, editing and mixing multitrack songs,
each track being associated with one or more plugins. For example, a guitar track will process the dry
sound that has been recorded, through a set of audio effects and/or a guitar amplifier simulation. A
keyboard track will record MIDI events that will be played by a virtual instrument. Each of these
(effects/instruments) could be a WebAudio Plugin. During the course of the WASABI ANR French
research project, we developed more than 70 open source plugins, some of them being now
commercialized by the I3S laboratory (part of French CNRS).
All these tools are available online and can be used for free (Amped Studio is a commercial application that
1
comes with a free plan, with some limitations), most are open-source (links are provided at the end of this
paper). Videos show some demonstrations of these tools and a step by step tutorial for creating WebAudio
2
plugins is also available online . Note that: as of early 2020, you need an up-to-date browser based on
3
1 Virtual Pedalboard: https://wasabi.i3s.unice.fr/dynamicPedalboard/#, Faust IDE with GUI Builder for WebAudio
plugins: https://mainline.i3s.unice.fr/fausteditorweb/dist/, AmpedStudio: https://ampedstudio.com/.
2 Videos: Virtual Pedalboard tutorial: https://www.youtube.com/watch?v=pe8zg8O-BFs, guitar amp sim:
https://www.youtube.com/watch?v=lIfm9ZMtG-I, GUI Builder and plugin publication: https://youtu.be/do-9mA42pNA,
also check all this YouTube channel: https://www.youtube.com/channel/UC0vZKRYdSHckcngkRMVyZRQ/videos
3 https://tinyurl.com/r223p7c
Chromium, or Firefox Nightly to run these applications. They are the only browsers with the required API
implementations, and polyfills for some of these APIs are not available or lack performance.
Fig 3. The Virtual Pedalboard host application discovers newly created plugins (using the REST APIs
of WAP servers). We can see on top left the plugin from Fig. 2.
FIg 4. The “Rack” plugin. Inspired by GarageBand’s pedal FX and Amp Sim manager.
Fig 5. The Amped Studio online digital workstation (DAW) supports WAP plugins.
Performance and compatibility concerns
Not all of the needed APIs are equally available on all browsers. The AudioWorklet API, part of the
WebAudio specification, required to run WebAssembly DSP code, is only publicly available on
Chromium-based browsers (in this implementation, the choice not to have the audio thread with real-time
priority can cause audio glitches in some circumstances, and is currently being questioned), while an
experimental version is available in Firefox Nightly, this time with a real-time approach. The WebMIDI
implementation is still only available in Chromium-based browsers for now. However, three commercial
online Digital Audio Workstations have been online for more than one year now, running successfully on
Chromium-based browsers, and will work on other browsers when implementations of the needed APIs will
become available (polyfills are not always available).
Security concerns
When a host application uses dynamically loaded plugins, it is obvious that some security precautions need to
be taken. The WAP standard comes with an SDK but also with online validation tools [1]. For example,
developers can check that a plugin complies with the WAP specification, can check its main features,
save/restore its state etc... Nevertheless, a publishing workflow is necessary before making a plugin available
in a commercial online host application such as Amped Studio. Tools are available to test plugins locally on
the developer's machine before submitting them to the internal validation process set up by host app
maintainers (plugins can be tried using an http server running on localhost, and can be seen by the host
webapp), then the plugin is submitted for further validation, including manual checks. The workflow is similar
to that of publishing mobile applications to app stores: automatic tools and manual validation are involved.
Conclusion
During the presentation we propose to demo these tools and give details on the design principles that guided
us so far. These WebAudio applications and tools show that online computer music applications can
convincingly compete with their native counterparts, making easier some use cases such as collaborative
music composition - as illustrated by SoundTrap being bought by Spotify and Amped Studio being offered to
the 300 million users of NetEase Cloud Music a Chinese music social network (sort of Deezer/Spotify in
4
China). This shows the maturity reached by recent Web APIs such as WebAudio, WebMIDI, WebAssembly,
WebComponents and others. And this also opened up commercial opportunities: open source WebAudio
software developed in the framework of academic projects such as the French ANR WASABI project, (guitar
amp simulators, FX pedals, Rack plugin) are now being marketed as addons for the Amped Studio DAW.
Acknowledgements
This work was supported by the French Research National Agency (ANR) and the WASABI team (contract
ANR-16-CE23-0017-01).
Open source resources
FAUST and FAUST IDE: https://faust.grame.fr/ https://faust.grame.fr/ide
WAP SDK: https://github.com/micbuffa/WebAudioPlugins
Virtual Pedalboard:
https://github.com/guizmo2000/Wasabi-Pedalboard/tree/dynamicLoading
Tube guitar amp simulation designer: https://mainline.i3s.unice.fr/AmpSim4/
Bibliography
[1] Buffa, M., Lebrun, J., Kleimola, J., Larkin, O. and Letz, S. “Towards an open Web Audio plugin standard”. In Companion
Proceedings (Developer’s track) of the The Web Conference 2018
(WWW 2018), Mar 2018, Lyon, France. hal-01721483
[2] Shihong Ren, Stéphane Letz, Yann Orlarey, Romain Michon, Dominique Fober, et al.. “FAUST online IDE: dynamically compile
and publish FAUST code as WebAudio Plugins”. WebAudio Conference 2019
, Dec 2019, Trondheim, Norway. hal-02366725
[3] Michel Buffa, Jerome Lebrun. “WebAudio Virtual Tube Guitar Amps and Pedal Board Design”. WebAudio Conference 2018
,
Sep 2018, Berlin, Germany. hal-01893781
[4] Michel Buffa, Jerome Lebrun. “Real time tube guitar amplifier simulation using WebAudio”. Web Audio Conference 2017 –
Collaborative Audio
#WAC2017", Queen Mary University of London, Aug 2017, London, United Kingdom. hal-01589229
[5] Michel Buffa, Jerome Lebrun. “Real-Time Emulation of a Marshall JCM 800 Guitar Tube Amplifier, Audio FX Pedals, in a
Virtual Pedal Board”. WWW2018 - TheWebConf 2018 : The Web Conference, 27th International World Wide Web Conference
,
Apr 2018, Lyon, France. 10.1145/3184558.3186973. hal-01721463
[6] Orlarey, Y., Fober, D., & Letz, S. 2004. “Syntactical and semantical aspects of Faust”. In Soft Computing, 8(9), 623-632.
[7] Letz, S., Orlarey, Y., and Fober, D. 2018. “Faust Domain Specific Audio DSP Language Compiler to WebAssembly”. In
Compagnion
Proc ot the Web Conference, International World Wide Web Conferences Steering Committee, Lyon France 2018
.
[8] Kleimola, J. and Larkin, O. 2015. “Web Audio modules”. In Proc. 12th Sound and Music Computing Conference (SMC 2015),
Maynooth, Ireland.
4 https://en.wikipedia.org/wiki/NetEase_Music
... This work resulted in many very well-evaluated emulations of these guitar amps as web-browser-based online apps relying essentially on the Web Audio API. The successive generations of simulations have been presented during many conferences, mostly at the Web Audio Conferences (WAC) [1,2,3,4,5,6,7,8,9,10] and at The Web Conferences (ACM WWW) [11,12,13,14]. With a consortium of other researchers and developers [15,16,17], this work also converged toward an open standard for plugins in Web Audio [5,6,7,8,9,11,12,13,14,18,19]. ...
... The successive generations of simulations have been presented during many conferences, mostly at the Web Audio Conferences (WAC) [1,2,3,4,5,6,7,8,9,10] and at The Web Conferences (ACM WWW) [11,12,13,14]. With a consortium of other researchers and developers [15,16,17], this work also converged toward an open standard for plugins in Web Audio [5,6,7,8,9,11,12,13,14,18,19]. This paper offers a digest of this long-term work with many details on the latest developments toward a versatile and comprehensive framework for the modular design of tube amp simulations. ...
... Using the Web Audio API, nodes can be connected to form an "audio graph", which enables developers to create more intricate audio effects such as the classic "Echo" (using DelayNode, BiquadFilterNode, GainNode), "Auto-Wah" (using BiquadFilterNode, OscillatorNode), "Chorus" (using multiple DelayNodes and OscillatorNodes for the modulation) or various types of Distortions (using GainNode, WaveShaperNode). These new building blocks can be assembled to create synthesizers, samplers, and other high-level audio effects or instruments [12]. With the introduction of the AudioWorklet node in the Web Audio API, it has become possible to write custom DSP code using a variety of programming languages such as C, C++, Rust, or domainspecific languages such as FAUST [26,27]. ...
Article
Full-text available
Since 2012 and the advent of the Web Audio API, the development of computer music applications within web browsers has become feasible. Recent improvements in this API and other advancements in web browser technology have significantly expanded the possibilities for new developments. Thanks to the WebAssembly and AudioWorklets APIs, code written in low-level languages (C/C++, Rust, . . .) or Domain Specific Languages can now be encapsulated and executed within browsers. Furthermore, using the Web Components API, it is now possible to produce interoperable Web Audio plugins. This article reviews seven years of work toward real-time, lightweight, low-latency, perceptually faithful browser-based simulations of tube guitar amplifiers. Summarizing previous conference presentations, this article provides details on the milestones and many issues that had to be dealt with during development to achieve these pioneering demonstrations of the potentialities of the Web Audio API. Most of the simulations are provided as open-source and freely available. Recently, some versions were licensed and integrated into a well-known commercial Digital Audio Workstation. During these years, several user tests have been conducted with professional guitarists who all positively evaluated these simulations and favorably benchmarked them against the few available commercial browser-based competitors.
... Since 2018 W3C Web standards have matured: the appearance of WebAssembly, stabilization of WebComponents, support for Au-dioWorklets [4] in the Web Audio API, and continued evolution of JavaScript have all helped make professional-grade, Web-based audio production a reality. In addition, commercial companies now offer digital audio workstations (DAW) on the Web which act as host Web applications and support plugins [3]. ...
... Multiple proposals for WebAudio plugin standardization arose since 2013 [1,[5][6][7]. We published a state of the art [1][2][3] on this domain which is still relevant today. ...
... Here are some examples of audio effects that could be built this way: echo (DelayNode, BiquadFilterNode, GainNode), auto-wah (BiquadFilterNode, OscillatorNode), chorus (multiple DelayNodes and OscillatorNodes for modulation), distortion (GainNode, WaveShaperNode), synthesizers, samplers etc. Existing DSP code in other languages such as C/C++ or coded using Domain Specific Languages such as FAUST, can be cross-compiled to Web Assembly and run in a single AudioWorklet node. Over the years, lots of high-level audio effects and instruments have been developed [4]. However, it is often necessary to chain such audio effects and instruments together (for example, in a guitarist's pedalboard) and during music composition/production in DAWs, multiple effects and instruments are used. ...
... A new version of the WebAudioModule (WAM) standard [16,17], which had been first released in 2015, was redesigned for reflecting recent technological developments. A URI can be used to fetch a WAM plugin from the web, and the current WebAudio context can be used to initialize it. ...
... Once compiled, the GUI has been customized within the GUI builder part of the IDE and published to remote WAP servers. They can now be tested online in host web applications such as our Virtual Pedalboard host presented in Figs 2,10 and 17. 39 Having the authoring tools as well as the deployment platform as pure Web applications facilitates the workflow and interoperability of these components [20]. We also think that the presented toolchain could be easily adapted to other plugin formats or audio DSP production tools. ...
Article
Full-text available
The development and porting of virtual instruments or audio effects on the Web is a hot topic. Several initiatives are emerging, from industry-driven ones (e.g., Propellerhead Rack Extension running on the Web) to more community-based open-source projects. Most of them aim at adapting existing code bases (usually developed in native languages like C/C++) as well as facilitating the use of existing audio Digital Signal Processing (DSP) languages and platforms. Our two teams previously presented an open format for WebAudio Plugins coined WAP. It aims at (i) improving the interoperability of audio/MIDI plugins developed using pure Web APIs, (ii) porting existing native code bases, or (iii) using Domain Specific Languages (DSL). In this paper, we present a solution based around FAUST DSL, its Web-based editor, and the integration of a plugin GUI editor allowing us to directly test, generate, and deploy WAP plugins. We also evoke our collaborative work: one team hatching and improving FAUST, the other working on the recreation of tube guitar amplifiers and pedalboards within Web browsers. So as to fully illustrate the FAUST online framework, a case study is detailed with complete workflow, from the FAUST DSP source code written and tested in a fully functional online editor to a self-contained plugin running in a separate host application.
Article
Full-text available
Cet article présente WAM Studio (Figure 1), une station de travail audio numérique (DAW) en ligne open source qui tire parti de plusieurs APIs et technologies standards du W3C, telles que Web Audio, WebAssembly, Web Components, Web Midi, Media Devices, etc. WAM Studio s’appuie également sur le standard Web Audio Modules (WAM), qui a été conçu pour faciliter le développement de plugins audio inter-opérables (effets, instruments virtuels, claviers virtuels de piano comme contrôleurs, etc.) sortes de "VSTs pour le Web". Les DAWs sont des logiciels riches en fonctionnalités et donc particulièrement complexes à développer en termes de conception, d’implémentation, de performances et d’ergonomie. Aujourd’hui, la majorité des DAWs en ligne sont commerciaux alors que les seuls exemples open source manquent de fonctionnalités (pas de prise en charge de plugins par exemple) et ne tirent pas parti des possibilités récentes offertes (comme WebAssembly). WAM Studio a été conçu comme un démonstrateur technologique pour promouvoir les possibilités offertes par les innovations récentes proposées par le W3C. L’article met en évidence certaines des difficultés que nous avons rencontrées (par exemple, les limitations dues aux environnements sandboxés et contraints que sont les navigateurs Web, la compensation de latence quand on ne peut pas connaître le hardware utilisé, etc.). Une démonstration en ligne, ainsi qu’un repository GitHub pour le code source sont disponibles. Wam Studio a également donné naissance à Attune, une version spéciale adaptée pour un projet de recherche en collaboration avec l’équipe MERI du CCRMA de Stanford, pour l’étude et la facilitation de l’écoute de musique multipiste par des personnes malentendantes équipées d’implants cochléaires.
Conference Paper
Full-text available
In this paper are exposed our latest experiments with the WebAudio API to design different types of gears for guitarists: real-time simulations of tube guitars amplifiers, fx pedals, and their integration in a virtual pedal board. We have studied different real guitar tube amps and created an interactive Web application for experimenting, validating and building different amp designs that can be run in browsers. Blind tests have been conducted with professional guitar players who compared positively our real-time, low-latency, realistic tube guitar amps simulations with state-of-the-art native equivalents. We also created a set of "virtual audio fx pedals" that implement popular audio effects such as flanger, chorus, overdrive, pitch shifter etc. These amps and pedals simulations can be packaged as "WebAudio plugins" and stored in plugin repositories (REST endpoints or local folders). We also developed a "host" application -a virtual pedal board- that allows us to scan repositories for plugins and to chain/assemble them.
Conference Paper
Full-text available
This paper presents a tube guitar amplifier simulation made with the WebAudio API, that reproduces the main parts of the Marshall JCM 800 amplifier schematics. Each stage of the real amp has been recreated (preamp, tone stack, reverb, power amp and speaker simulation, and we added an extra multiband EQ). The “classic rock” amp simulation we built has been used in real gigs and can be compared with some native amp simulation both in terms of latency, sound quality, dynamics and comfort of the guitar play. Unfortunately, as of today, low latency can be achieved only with certain configurations, due to audio driver limitations of current browsers on certain operating systems. The paper discusses the latency problems encountered with WebAudio, common traps, current limitations, and proposes some solutions.
Conference Paper
Full-text available
\beginabstract This paper demonstrates how FAUST, a functional programming language for sound synthesis and audio processing, can be used to develop efficient audio code for the Web. After a brief overview of the language, its compiler and the architecture system allowing to deploy the same program as a variety of targets, the generation of WebAssembly code and the deployment of specialized WebAudio nodes will be explained. Several use cases will be presented. Extensive benchmarks to compare the performance of native and WebAssembly versions of the same set of DSP have be done and will be commented. \endabstract
Conference Paper
Full-text available
Web Audio is a recent W3C API that brings the world of computer music applications to the browser. Although developers have been actively using it since the first beta implementations in 2012, the number of web apps built using Web Audio API cannot yet compare to the number of commercial and open source audio software tools available on native platforms. Many of the sites using this new technology are of an experimental nature or are very limited in their scope. While JavaScript and Web standards are increasingly flexible and powerful, C and C++ are the languages most often used for real-time audio applications and domain specific languages such as FAUST facilitate rapid development with high performance. Our work aims to create a continuum between native and browser based audio app development and to appeal to programmers from both worlds. This paper presents our proposal including guidelines and proof of concept implementations for an open Web Audio plug-in standard - essentially the infrastructure to support high level audio plug-ins for the browser.
Conference Paper
Full-text available
The ANR project WASABI will last 42 months and consists in developing a 2 million songs database with interactive WebAudio enhanced client applications. Client applications target composers, music schools, sound engineering schools, musicologists, music streaming services and journalists. In this paper, we present a virtual pedal board (a set of chainable audio effects on the form of "pedals"), and a guitar tube amplifier simulation for guitarists, that will be associated with songs from the WASABI database. Music schools and music engineering schools are interested in such tools that can be run in a Web page, without the need to install any further software. Take a classic rock song: isolate the guitar solo, study it, then mute it and play guitar real-time along the other tracks using an online guitar amplifier that reproduces the real guitar amp model used in the song, with its signature sound, proper dynamic and frequency response. Add some audio effects such as a reverberation, a delay, a flanger, etc. in order to reproduce Pink Floyd's guitar sound or Eddie Van Halen famous "Brown Sound". Learn interactively, guitar in hands, how to fine tune a compressor effect, or how to shape the sound of a tube guitar amp, how to get a "modern metal" or a "Jimi Hendrix" sound, using only your Web browser.
Article
Full-text available
This paper presents some syntactical and semantical aspects of FAUST (Functional AUdio STreams), a programming language for real-time sound processing and synthesis. The programming model of FAUST combines two approaches: functional programming and block-diagrams composition. It is based on a block-diagram algebra. It as a well defined formal semantic and can be compiled into efficient C/C++ code.
FAUST online IDE: dynamically compile and publish FAUST code as WebAudio Plugins
  • Stéphane Shihong Ren
  • Yann Letz
  • Romain Orlarey
  • Dominique Michon
  • Fober
Shihong Ren, Stéphane Letz, Yann Orlarey, Romain Michon, Dominique Fober, et al.. "FAUST online IDE: dynamically compile and publish FAUST code as WebAudio Plugins". WebAudio Conference 2019, Dec 2019, Trondheim, Norway. ⟨hal-02366725⟩