Maestro is a tool for processing library SoundCloud Web broadcast, it successfully processed tens of millions of players every day at soundcloud.com, SoundCloud mobile sites, web plug-in, Chromecast and Xbox applications. Today, we are considering open source, this blog will introduce our technology achievements to date in the development process Maestro made. Panda thank former live front-end technology experts Jiang Yuqing translation and proofreading this article.
Statement / Tom Jenkinson
Translation & technical reviewer / Jiang Yuqing
Supportive
In SoundCloud, we hope to support all modern web browsers, mobile browser and IE 11. Our goal is to provide the best playback experience using the functionality provided by the browser.
Media stream
We currently support three decoders media stream:
mp3
opus
aac
Our main agreement is HLS (HTTP Live Streaming). This means that the audio content will be cut into fragments, we have a separate file (playlist), which contains the URL for all segments, as well as their corresponding time in the audio content. You can find more information about HLS here.
Content provided by the browser
We use your browser's audio label media source extension (MSE) and the Web Audio API.
We need a browser that supports at least audio label, with streaming media decoding and playback capabilities. MSE and the Web Audio API is the best experience necessary.
When the Web Audio API or MSE is lost or an error occurs during playback, we can downgrade to normal.
We will use a little more about our MSE and content Web Audio API, but first, let's look at the audio tag is what we did.
audio
If the browser supports decoding, you can obtain the URL of an audio file and play. It will notify the codec in the header of the response content-type, which provides an API, which is used to control playback and to determine whether the browser supports decoding:
Extended media source
Use only audio tag, the browser can do all the work behind the scenes, but you do not have access to its underlying buffer.
Use MSE, we can create a buffer for the browser to support the decoder. Then we can deal with their own media download and attach it to the buffer. This means we can be optimized, such as: pre-loaded, which we when you click the play button, it is stored in memory, we believe that pre-download the first few seconds of the audio file that you will play. Then when you click play, we add this data directly from memory to buffer without having to obtain from the network:
const audio = document.createElement('audio');
const mse = new MediaSource()
const url = URL.createObjectURL(mse)
audio.src = url
audio.play()
mse.addEventListener('sourceopen', () => {
// 'audio/mpeg' for mp3
const buffer = mse.addSourceBuffer('audio/mpeg');
buffer.mode = 'sequence';
const request = new Request('http://example.invalid/segment0.mp3');
fetch(request).then((response) => response.arrayBuffer()).then((data) => {
buffer.appendBuffer(data);
});
});
Web Audio API
Web Audio API是这里提到的最新的API。当您播放，暂停或搜索时，我们会使用此API的一小部分来快速淡入淡出。这使得播放体验更加的爽快、播放/暂停不那么突然：
const audio = document.createElement('audio');
const context = new AudioContext();
const sourceNode = context.createMediaElementSource(audio);
const gainNode = context.createGain();
sourceNode.connect(gainNode);
gainNode.connect(context.destination);
audio.src = 'http://example.invalid/something.mp3';
audio.play();
// Schedule fade out.
gainNode.gain.linearRampToValueAtTime(0, context.currentTime + 1);
Maestro的目标
简单的API
插件架构
易于检测功能
类型安全
支持所有主流浏览器
处理浏览器实现中的差异和错误
优异的性能
能够预加载
尽可能地响应
可配置的缓冲区长度和缓存大小
能够在具有内存受限的设备上工作，如Chromecast
检测
提供错误数据和性能数据，对其进行监控，以检测错误并进行改进
技术栈
TypeScript
Lerna
Yarn
WebPack
API
Maestro包含许多包。核心包提供了一个抽象BasePlayer类，它提供了播放器API。它将任务委派给特定的实现，外部通信通过BasePlayer。可以通过player 方法检索最新状态，并且在有任何更改时通知用户。
例如，该play()方法返回Promise可以解析或拒绝。这BasePlayer将告知是县城何时应该播放或暂停，实现层将告知BasePlayer实际播放的时间。每个播放器实现都与实际play()方法分离。这也意味着isPlaying()可以完全处理方法和相应的更新BasePlayer。另一个例子是getPosition()，除了通知实现层播放时间，除非正在seek，在这种情况下BasePlayer将返回请求的时间点。这意味着时间getPosition()总是有意义的，用户在seek时可以保证它不会跳转，并覆盖它。
播放器实现包含在单独的包中，并且它们都扩展BasePlayer。我们目前有以下播放器：
HTML5Player - 这是最简单的播放器。它采用URL和MIME类型，它们直接传递给媒体元素。
HLSMSEPlayer- 这扩展了HTML5Player，它需要一个Playlist对象来供段数据。该播放器使用MSE。
ChromecastPlayer - 此播放器是一个控制Chromecast的代理。
ProxyPlayer - 此播放器可以控制另一个播放器以便随时切换。它还具有一些提供新播放器同步相关的配置。该播放器的一个好处是，它可以在真正的播放器还没有的时候同步提供给应用程序。然后，一旦真实播放器可用，其状态将被同步以匹配代理。其他一些用例是在Chromecast上播放和本地播放，或切换质量。该应用程序只需与一个播放器进行交互，切换可以在幕后进行。
状态管理和事件
在Maestro中，有很多播放状态需要管理，它们大部分都包含在内部BasePlayer。用户还想知道某些部分的状态何时发生变化，有时会通过执行其他播放器操作来对变化作出反应。当我们在单个线程上运行时，这会带来一些复杂性。有时我们还会以原子方式（跨多个函数）更新状态的几个部分。例如：如果用户跳转到媒体的结尾，我们也想要将ended标志更新为true。更新ended标志有关的逻辑，与代码中的查找逻辑无关，但跳转状态和结束状态的更新应该在API中一起发生。
为实现这一目标，我们构建了一个名为的组件StateManager，它使我们能够：
在调用之前更新函数的多个部分，以通知用户更改。
在播放器调用堆栈的末尾通知用户状态更改，以便他们与播放器的任何交互不会因此而在调用堆栈中交错。（例如，执行工作然后触发事件，而不是触发事件然后执行工作。）
StateManager
StateManager维护一个状态对象。对该对象的所有更改都是使用update()方法进行的，并且可以提供回调，然后在update()最后通知回调发生的任何状态更改。这些调用可以嵌套：
type ChangesCallback<State> = (changes: Readonly<Partial<State>>, state: Readonly<State>) => void;
type Control = {
remove: () => boolean;
};
type Subscriber<State> = {
callback: ChangesCallback<State>,
localState: State
};
class StateManager<State extends { [key: string]: Object | null }> {
private _state: State;
private _subscribers: Array<Subscriber<State>> = [];
private _updating = false;
constructor(initialState: State) {
this._state = clone(initialState);
// ...
}
public update(callback: (state: State) => void): void {
const wasUpdating = this._updating;
this._updating = true;
try {
callback(this._state);
} catch(e) {
// error handling...
}
if (!wasUpdating) {
this._updating = false;
this._afterUpdate();
}
}
public subscribe(callback: ChangesCallback<State>, skipPast = true): Control {
// ...
}
private _afterUpdate(): void {
this._subscribers.slice().forEach((subscriber) => {
const diff = this._calculateDiff(subscriber.localState);
// We always recalculate the diff just before calling a subscriber,
// which means that the state is always up to date at the point when
// the subscriber is called.
if (Object.keys(diff).length) {
subscriber.localState = clone(this._state);
deferException(() => subscriber.callback(diff, subscriber.localState));
}
});
}
private _calculateDiff(compare: State): Readonly<Partial<State>> {
// ...
}
}
示例用法
type OurState = { a: number, b: string, c: boolean, d: number };
const stateManager = new StateManager<OurState>({
a: 1,
b: 'something',
c: true,
d: 2
});
stateManager.subscribe(({ a, b, c, d }) => {
// On first execution:
// a === 2
// b === 'something else'
// c === false
// d === undefined
// On second execution:
// a === undefined
// b === undefined
// c === undefined
// d === 3
updateD();
});
stateManager.subscribe(({ a, b, c, d }) => {
// a === 2
// b === 'something else'
// c === false
// d === 3
});
doSomething();
function doSomething() {
stateManager.update((state) => {
state.a = 2;
updateB();
state.c = false;
});
}
function updateB() {
stateManager.update((state) => {
state.b = 'something else';
});
}
function updateD() {
stateManager.update((state) => {
state.d = 3;
});
}
请注意，第一个订阅回调将执行两次，第二个订阅也只执行一次，并且只执行最新状态（即d === 3）。
另请注意，我们不会获得嵌套调用堆栈，因为回调只在工作完成后才会执行。
浏览器限制
不幸的是，不同的浏览器具有不同的编解码器支持（也可能取决于操作系统）和不同的容器需求。
例如，Chrome支持MSE中的原始MP3文件，但Firefox要求MP3位于MP4容器中。这意味着在Firefox中，我们需要将我们下载的MP3打包到浏览器中的MP4中。其他编解码器具有类似的复杂性。
有bug也是不可避免的。为支持在安全的方式下，处理各种媒体的媒体处理管道，并且不破坏Web浏览器的向后兼容性，这是一项艰巨的任务！幸运的是，Maestro有能够处理不同浏览器中各种错误的变通方法，其中一些在版本之间有所不同。
浏览器之间的自动播放策略也不同，这意味着我们目前必须在播放器之间共享媒体元素。这增加了复杂性，因为当元素的源被更改时，仍然会在之后的短时间内为前一个源发出事件，这意味着我们必须在尝试使用它之前等待事件“清空”，并且我们必须保持跟踪同时请求的所有内容。Maestro的HTML5Player通过使用provideMediaElement(mediaEl)和revokeMediaElement()让这变得简单。这允许您在运行时在播放器之间移动媒体元素。当播放器没有媒体元素时，播放器就会暂停。
测试
在BasePlayer和播放器的实现是通过单元测试和集成测试覆盖：我们采用Mocha，Sinon，karma，以及mocha-screencast-reporter。后者非常适合远程查看测试的运行进度。
确保API的行为正确，该BasePlayer自身目前拥有超过700次测试。例如，测试检查play()实现是否正在播放时解析了promise。一个测试play()如果在播放请求完成之前播放器被释放，则另一个测试会被拒绝并返回正确的报错。还有一些测试可以检查播放器是否在检测到不一致时报错。 例如，一个播放器实现在BasePlayer从未请求过seek操作时，无法完成seek请求。
我们还使用SauceLabs在各种浏览器和浏览器版本（包括Chrome和Firefox beta）上运行所有测试。这需要几个小时才能完成，因此我们测试了各主流浏览器，我们在发布之前测试所有内容。我们还每周运行所有测试，以确保新浏览器版本不会出现任何问题。这样做，曾有一次高亮显示了Firefox beta中的Web Audio错误，这会导致播放在前几秒后停止。
渐进式流媒体（使用fetch()API）
我们最近添加了对渐进式流式传输的支持（在支持的浏 这意味着在我们处理它并将其附加到缓冲区之前不必等待整个段被下载，我们能够在数据到达时处理数据，这意味着我们能够在段下载之前开始播放已完成。
这是通过fetch()API（以及moz-chunked-arraybuffer在Firefox中）实现的，它在下载时仍提供小部分数据：
fetch(new Request(url)).then(({ body }) => {
return body.pipeTo(new WritableStream({
write: (chunk) => {
console.log('Got part', chunk);
},
abort: () => {
console.log('Aborted');
},
close: () => {
console.log('Got everything');
}
}));
});
在我们添加渐进式流式传输之前，如果下载失败，我们只会重试它，这个逻辑非常独立。使用渐进式流式传输更为复杂，因为如果下载部分失败，整个管道已经开始处理数据。我们决定在错误时重试请求并丢弃我们已经看到的所有字节。如果重试失败，那么我们就能够在管道中产生报错。
这也带来了更多的复杂性。之前，我们知道每个段包含完整数量的有效音频单元，这意味着管道的不同部分可以做出某些响应。现在，每个数据部分都可以包含一小部分音频单元，因此我们需要能够检测到何时发生这种情况，并保留和等待一个完整单元到达的缓冲区。
下一步是什么？
Since June 2017 we began running Maestro, but rarely played bad feedback problem. We can monitor real-time performance and error, and in the event of an error, we were able to retrieve the play log, which helps debugging.
We are looking for the next target Maestro, that is your participation: Let us know how you will use it, as well as the function you want to see: D
If you have any questions about this post, or if you notice any playback problems on soundcloud.com;), please contact us!
Jiang Yuqing additional information
MSDN https://developer.mozilla.org/zh-CN/docs/Web/API/AudioContext some explanation about AudioContext on AudioContext, it will eat a little performance.
Progressive streaming media, is what we call live streaming, it sometimes involves a complete segment is not a problem, we must wait for the complete fragment, before you write the code has done deal. https://github.com/xiongmaotv/open-mccree/blob/f8491e33770c59fe6288f1a05daf8375d4f01820/packages/mccree-core-loaderbuffer/src/index.js#L46
State processing repeatedly mentioned in the article, because many media player is an asynchronous method, especially direct and there are differences in different browsers, such as stop downloading cancel a Promise, in asynchronous in chrome, FireFox are not.
LiveVideoStack recruitment
LiveVideoStack is recruiting editors / reporters / operators, together with the world's leading multimedia and technical experts and LiveVideoStack younger partners, to promote eco-development of multimedia technology. Please understand post information directly employed in the BOSS search "LiveVideoStack", or the exchange of research and editor of the package through the micro-letter "Tony_Bao_".
[Click to read the original ] to learn more about the General Assembly lecturer and content information.