Simple arrangement of loading and playback of three.js (webgl) basic audio of Three

Simple arrangement of loading and playback of three.js (webgl) basic sounds of Three

content

Simple arrangement of loading and playback of three.js (webgl) basic sounds of Three

1. Brief introduction

2. Three elements to realize audio playback in Three

3. THREE.Audio general player (non-position playback)

4. THREE.PositionalAudio position player (a more realistic sound player, the louder the sound when you walk in, the lower the sound when you leave)

5. AudioAnalyser audio parser (obtain sound data, perform visual sound operations, etc.)

---

 

1. Brief introduction

Some knowledge developed by Three js is convenient to encounter similar problems in the later period, and can be checked and used in time.

This section introduces that sound and music in three.js ( webgl ) are indispensable, audio loading and playback, here is a brief introduction, if there are shortcomings, please point out, or if you have a better method, welcome to leave a message.

2. Three elements to realize audio playback in Three

To achieve sound playback in Three js, generally you need to use three elements (three parts) in Three:

1. AudioListener, audio listener, that is, the ears that seem to listen to the sound

2. Audio, which is what we call the sound source every day

3, AudioLoader, audio loader, used to load the audio files we need to play

Threejs provides a series of audio related APIs: Audio Audio , Positional Audio PositionalAudio , Listener AudioListener , Audio Analyzer AudioAnalyser , Audio Loader AudioLoader .

Threejs classes such as audio Audioand positional audio PositionalAudioare essentially encapsulations of the native Web Audio API .

AudioListener uses a virtual listener to represent all positional and non-positional related sound effects in the scene.
A three.js program usually creates an AudioListener. It is a required parameter of the constructor of audio entities, such as Audio and PositionalAudio.
In most cases, The listener object is a sub-object of the camera. The camera's 3D transformation represents the listener's 3D transformation.

 

3. THREE.Audio General player (non-positional playback)

THREE.AudioLoad a piece of audio for playback through a general player . The audio playback effect is not affected by the specific location. Generally, it can be used for the global background music of a 3D scene .

// 创建一个 AudioListener 并将其添加到 camera 中
const listener = new THREE.AudioListener();
camera.add( listener );

// 创建一个全局 audio 源
const sound = new THREE.Audio( listener );

// 加载一个 sound 并将其设置为 Audio 对象的缓冲区
const audioLoader = new THREE.AudioLoader();
audioLoader.load( 'sounds/ambient.ogg', function( buffer ) {
	sound.setBuffer( buffer );
	sound.setLoop( true );
	sound.setVolume( 0.5 );
	sound.play();
});

4. THREE.PositionalAudio Position player (a more realistic sound player, the louder the sound when you walk in, the smaller the sound when you leave)

In real life, the effect of hearing the sound is affected by the position and angle of the sound source relative to the listener. The position of the audio source changes, and the sound heard changes, such as the volume level. Threejs provides an THREE.AudioAPI positional audio THREE.PositionalAudiothat is different from non-positional audio. The audio playback effect created by positional audio THREE.PositionalAudiocan imitate the sound effects of people hearing sound sources in different positions in nature.

// create an AudioListener and add it to the camera
const listener = new THREE.AudioListener();
camera.add( listener );

// create the PositionalAudio object (passing in the listener)
const sound = new THREE.PositionalAudio( listener );

// load a sound and set it as the PositionalAudio object's buffer
const audioLoader = new THREE.AudioLoader();
audioLoader.load( 'sounds/song.ogg', function( buffer ) {
	sound.setBuffer( buffer );
	sound.setRefDistance( 20 );
	sound.play();
});

// create an object for the sound to play from
const sphere = new THREE.SphereGeometry( 20, 32, 16 );
const material = new THREE.MeshPhongMaterial( { color: 0xff2200 } );
const mesh = new THREE.Mesh( sphere, material );
scene.add( mesh );

// finally add the sound to the mesh
mesh.add( sound );

5. AudioAnalyser audio parser (obtain sound data, perform visual sound operations, etc.)

Create AudioAnalyser object, use AnalyserNode to analyze audio data..

Code examples based on Threejs   GitHub - mrdoob/three.js: JavaScript 3D Library.

<!DOCTYPE html>
<html lang="en">
	<head>
		<title>three.js webaudio - visualizer</title>
		<meta charset="utf-8">
		<meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0">
		<link type="text/css" rel="stylesheet" href="main.css">

		<script id="vertexShader" type="x-shader/x-vertex">

			varying vec2 vUv;

			void main() {

				vUv = uv;

				gl_Position = vec4( position, 1.0 );

			}

		</script>

		<script id="fragmentShader" type="x-shader/x-fragment">

			uniform sampler2D tAudioData;
			varying vec2 vUv;

			void main() {

				vec3 backgroundColor = vec3( 0.125, 0.125, 0.125 );
				vec3 color = vec3( 1.0, 1.0, 0.0 );

				float f = texture2D( tAudioData, vec2( vUv.x, 0.0 ) ).r;

				float i = step( vUv.y, f ) * step( f - 0.0125, vUv.y );

				gl_FragColor = vec4( mix( backgroundColor, color, i ), 1.0 );

			}

		</script>

	</head>
	<body>
		<div id="overlay">
			<button id="startButton">Play</button>
		</div>
		<div id="container"></div>
		<div id="info">
			<a href="https://threejs.org" target="_blank" rel="noopener noreferrer">three.js</a> webaudio - visualizer<br/>
			music by <a href="http://www.newgrounds.com/audio/listen/376737" target="_blank" rel="noopener">skullbeatz</a>
		</div>

		<!-- Import maps polyfill -->
		<!-- Remove this when import maps will be widely supported -->
		<script async src="https://unpkg.com/[email protected]/dist/es-module-shims.js"></script>

		<script type="importmap">
			{
				"imports": {
					"three": "../build/three.module.js"
				}
			}
		</script>

		<script type="module">

			import * as THREE from 'three';

			let scene, camera, renderer, analyser, uniforms;

			const startButton = document.getElementById( 'startButton' );
			startButton.addEventListener( 'click', init );

			function init() {

				const fftSize = 128;

				//

				const overlay = document.getElementById( 'overlay' );
				overlay.remove();

				//

				const container = document.getElementById( 'container' );

				renderer = new THREE.WebGLRenderer( { antialias: true } );
				renderer.setSize( window.innerWidth, window.innerHeight );
				renderer.setClearColor( 0x000000 );
				renderer.setPixelRatio( window.devicePixelRatio );
				container.appendChild( renderer.domElement );

				scene = new THREE.Scene();

				camera = new THREE.Camera();

				//

				const listener = new THREE.AudioListener();

				const audio = new THREE.Audio( listener );
				const file = './sounds/376737_Skullbeatz___Bad_Cat_Maste.mp3';

				if ( /(iPad|iPhone|iPod)/g.test( navigator.userAgent ) ) {

					const loader = new THREE.AudioLoader();
					loader.load( file, function ( buffer ) {

						audio.setBuffer( buffer );
						audio.play();

					} );

				} else {

					const mediaElement = new Audio( file );
					mediaElement.play();

					audio.setMediaElementSource( mediaElement );

				}

				analyser = new THREE.AudioAnalyser( audio, fftSize );

				//

				const format = ( renderer.capabilities.isWebGL2 ) ? THREE.RedFormat : THREE.LuminanceFormat;

				uniforms = {

					tAudioData: { value: new THREE.DataTexture( analyser.data, fftSize / 2, 1, format ) }

				};

				const material = new THREE.ShaderMaterial( {

					uniforms: uniforms,
					vertexShader: document.getElementById( 'vertexShader' ).textContent,
					fragmentShader: document.getElementById( 'fragmentShader' ).textContent

				} );

				const geometry = new THREE.PlaneGeometry( 1, 1 );

				const mesh = new THREE.Mesh( geometry, material );
				scene.add( mesh );

				//

				window.addEventListener( 'resize', onWindowResize );

				animate();

			}

			function onWindowResize() {

				renderer.setSize( window.innerWidth, window.innerHeight );

			}

			function animate() {

				requestAnimationFrame( animate );

				render();

			}

			function render() {

				analyser.getFrequencyData();

				uniforms.tAudioData.value.needsUpdate = true;

				renderer.render( scene, camera );

			}

		</script>

	</body>
</html>