Examples of WebRTC audio and video capture and playback and MediaStream media stream analysis
Table of contents
- Sample code - open the camera and microphone at the same time, and display the screen and play the captured sound on the page
- API analysis
- mediaDevices
- MediaStream media stream
1. Sample code - open the camera and microphone at the same time, and display the screen and play the captured sound on the page
- the code
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>WebRTC Demo</title>
</head>
<body>
<video id="local-video" autoplay playsinline></video>
<button id="showVideo">打开音视频</button>
</body>
<script>
const constraints = {
audio: true,
video: {
width: 640, height: 480}
}
// 处理打开摄像头成功
function handleSuccess(mediaStream) {
const video = document.querySelector("#local-video");
video.srcObject = mediaStream;
}
// 异常处理
function handleError(error) {
console.error("getUserMedia error: " + error)
}
function onOpenAV(e) {
navigator.mediaDevices.getUserMedia(constraints).then(handleSuccess).catch(handleError);
}
document.querySelector("#showVideo").addEventListener("click", onOpenAV)
</script>
</html>
- Effect
2. API Analysis
1. mediaDevices
- mediaDevices is a Navigator read-only property that returns a MediaDevices object that provides connection access to media input devices such as cameras and microphones, including screen sharing.
- grammar:
var mediaDevices = navigator.mediaDevices;
- MediaDevices is a singleton object and members of this object can be used directly, for example by calling navigator.mediaDevices.getUserMedia().
2. MediaStream media stream
- navigator.mediaDevices.getUserMedia() will prompt the user to give permission to use the media input, and the media input will generate a MediaStream (media stream), which is the carrier of information and represents the content stream of a media device.
- Media streams can be collected, transmitted, and played. Usually, a media stream contains multiple media tracks, such as audio tracks and video tracks.
- Media streams are managed using the MediaStream interface. Usually, there are several ways to obtain media streams.
a. Get the stream object from the camera or microphone.
b. Get the stream object from the screen share.
c. Get the stream object from the canvas (HTMLCanvasElement) content.
d. Get the stream object from the media element (HTMLMediaElement). - The media streams obtained by the above methods can be transmitted through WebRTC and shared among multiple peers.
- Return a Promise object, and resolve will call back a MediaStream object after success.
- If the user denies the permission, or the required media source is not available, the promise will reject and call back a PermissionDeniedError or NotFoundError.
- Common usage:
navigator.mediaDevices.getUserMedia(constraints)
.then(function(stream) {
/* 使用这个 stream stream */
})
.catch(function(err) {
/* 处理 error */
});
- Definition of MediaStream
interface MediaStream : EventTarget {
constructor();
constructor(MediaStream stream);
constructor(sequence<MediaStreamTrack> tracks);
readonly attribute DOMString id;
sequence<MediaStreamTrack> getAudioTracks();
sequence<MediaStreamTrack> getVideoTracks();
sequence<MediaStreamTrack> getTracks();
MediaStreamTrack? getTrackById(DOMString trackId);
void addTrack(MediaStreamTrack track);
void removeTrack(MediaStreamTrack track);
MediaStream clone();
readonly attribute boolean active;
attribute EventHandler onaddtrack;
attribute EventHandler onremovetrack;
};
1. MediaStream property
1. active read-only
- Returns the state of MediaStream, the type is Boolean, true means it is active, false means it is inactive.
2. id is read-only
- Returns the UUID of the MediaStream, the type is string and the length is 36 characters.
2. MediaStream method
1. addTrack() method: add a new media track to the media stream.
stream.addTrack(track);
参数：Track，媒体轨道，类型为MediaStreamTrack。
返回值：无。
2. clone() method: returns a copy of the current media stream, and the copy has a different and unique identifier.
const newstream = stream.clone();
// sameId为false
const sameId = newstream.id === stream.id? true : false
参数：无。
返回值：一个新的媒体流对象。
3. getAudioTracks() method: returns an array of media track objects whose media type is audio, and whose member type is MediaStreamTrack.
- Note that the order of the array is indeterminate and may vary from call to call.
const mediaStreamTracks = mediaStream.getAudioTracks()
参数：无。
返回值：mediaStreamTracks，媒体轨道对象数组，如果当前媒体流没有音频轨道，则返回数组为空。
- Example: Use the getUserMedia() method to obtain the media stream containing video and audio tracks. If the call is successful, attach the media stream to the < >
videoelement, then set the timer, and call the getAudioTracks() method to obtain all audio tracks after 5 seconds, and finally stop Play the first audio track.
navigator.mediaDevices.getUserMedia({
audio: true, video: true})
.then(mediaStream => {
document.querySelector('video').srcObject = mediaStream;
// 5s后，停止播放第一个音频轨道
setTimeout(() => {
const tracks = mediaStream.getAudioTracks()
tracks[0].stop()
}, 5000)
})
4. getVideoTracks() method: returns an array of media track objects whose kind attribute value is video, and the type of the media track object is MediaStream Track.
- Note that the order of the objects in the array is indeterminate and may vary from call to call.
const mediaStreamTracks = mediaStream.getVideoTracks()
参数：无。
返回值：mediaStreamTracks是媒体轨道对象数组。如果当前媒体流没有视频轨道，则返回数组为空。
- Example: The getUserMedia() method gets the video stream, if the call is successful, attach the media stream to the <
video> element, then get the first video track and capture a picture from the video track.
navigator.mediaDevices.getUserMedia({
video: true})
.then(mediaStream => {
document.querySelector('video').srcObject = mediaStream;
const track = mediaStream.getVideoTracks()[0];
// 截取图片
const imageCapture = new ImageCapture(track);
return imageCapture;
})
5. getTrackById() method: returns the track object with the specified ID.
- If no parameter is provided, or the ID value is not matched, null is returned; if there are multiple tracks with the same ID, the method returns the first track that matches.
const track = MediaStream.getTrackById(id);
参数：id，类型为字符串。
返回值：如果输入参数id与MediaStreamTrack.id匹配，则返回相应的MediaStream-Track对象，否则返回null。
- Example: Get the media track with the specified ID and apply constraints to adjust the volume to 0.5.
stream.getTrackById("primary-audio-track").applyConstraints({
volume: 0.5 });
6. getTracks() method: returns an array of all media track objects, including all video and audio tracks.
- The order of the objects in the array is indeterminate and may vary from call to call.
const mediaStreamTracks = mediaStream.getTracks()
参数：无。
返回值：媒体轨道对象数组。
- Use the getUserMedia() method to get the stream containing the video track. If the call is successful, attach the stream to the <
video> element, then set the timer, get all the media tracks after 5s, and stop playing the first media track (that is, the video track) .
navigator.mediaDevices.getUserMedia({
audio: false, video: true})
.then(mediaStream => {
document.querySelector('video').srcObject = mediaStream;
// 5s后，停止播放第一个媒体轨道
setTimeout(() => {
const tracks = mediaStream.getTracks()
tracks[0].stop()
}, 5000)
})
3. MediaStream event
1. addtrack event: This event is triggered when a new media track (MediaStreamTrack) is added, corresponding to the event handler onaddtrack
- Note that this event will only be triggered under the following conditions, and will not be triggered if the MediaStream.addTrack() method is actively called.
- RTCPeerConnection renegotiation.
- HTMLMediaElement.captureStream() returns a new media track.
- Example: When a new media track is added to the media stream, display the type and label of the new media track.
// event类型为MediaStreamTrackEvent
// event.track类型为MediaStreamTrack
stream.onaddtrack = (event) => {
let trackList = document.getElementById("tracks");
let label = document.createElement("li");
label.innerHTML = event.track.kind + ": " + event.track.label;
trackList.appendChild(label);
};
- In addition, you can also use the addEventListener() method to listen to the event addtrack.
2. removetrack event: This event is triggered when a media track is removed, corresponding to the event handler onremovetrack
- Note that this event will only be triggered in the following situations, and will not be triggered if the MediaStream.removeTrack() method is actively called.
- RTCPeerConnection renegotiation.
- HTMLMediaElement.captureStream() returns a new media track.
- Example: When a media track is deleted from a media stream, record the media track information.
// event类型为MediaStreamTrackEvent
// event.track类型为MediaStreamTrack
stream.onremovetrack = (event) => {
let trackList = document.getElementById("tracks");
let label = document.createElement("li");
label.innerHTML = "Removed: " + event.track.kind + ": " + event.track.label;
trackList.appendChild(label);
};
- In addition, you can also use the addEventListener() method to listen to the event removetrack.
4. Parameter constraints
-
constraints, as a MediaStreamConstraints object, specifies the requested media type and corresponding parameters.
-
The constraints parameter is a MediaStreamConstraints object that contains two members, video and audio, to describe the requested media type.
- At least one type or both must be specified.
- If the browser cannot find the specified media type or cannot meet the corresponding parameter requirements, the returned Promise object will be in the rejected [failed] state, and NotFoundError will be used as the parameter of the rejected [failed] callback.
-
The following requests both audio and video without any parameters:
{
audio: true, video: true }
-
If true is set for a certain media type, the resulting stream needs to have that type of track in it. If one of them cannot be obtained for some reason, getUserMedia() will raise an error.
-
When the user's camera and microphone information cannot be accessed due to privacy protection reasons, the application can use the additional constraints parameter to request the camera and microphone capabilities it needs or wants.
-
The following demonstrates that the app wants to use a camera resolution of 1280x720:
{
audio: true,
video: {
width: 1280, height: 720 }
}
- The browser will try to satisfy this request parameter, but if it cannot accurately meet the parameter requirements in this request or the user chooses to override the parameter in the request, it may return other resolutions.
- When forcing a specific size, you can use the keywords min, max or exact (that is, min == max).
- The following parameters indicate that a minimum resolution of 1280x720 is required.
{
audio: true,
video: {
width: {
min: 1280 },
height: {
min: 720 }
}
}
- If the camera does not support the requested or higher resolution, the returned Promise will be in rejected state, NotFoundError will be passed as the parameter of the rejected callback, and the user will not be prompted for authorization.
- The reason for the different performance is that compared with the simple request value and the ideal keyword, the keywords min, max, and exact have intrinsic coerciveness, such as:
{
audio: true,
video: {
width: {
min: 1024, ideal: 1280, max: 1920 },
height: {
min: 776, ideal: 720, max: 1080 }
}
}
- When the request contains an ideal value, this value has a higher weight, meaning that the browser will first try to find the setting or camera (if the device has more than one camera) that is closest to the specified ideal value.
- A simple request value can also be understood as an ideal value for the application, so our first request for specifying a resolution can also be written as follows:
{
audio: true,
video: {
width: {
ideal: 1280 },
height: {
ideal: 720 }
}
}
- Not all constraints are numbers. For example, on a mobile device, the following example indicates that the front camera should be used first (if available):
{
audio: true, video: {
facingMode: "user" } }
- To force the use of the rear camera, please use:
{
audio: true, video: {
facingMode: {
exact: "environment" } } }
- In some cases, such as when using a limited bandwidth transport over WebRTC, a low frame rate may be more appropriate.
{
video: {
frameRate: {
ideal: 10, max: 15 } } };
5. Return value
var promise = navigator.mediaDevices.getUserMedia(constraints);
- Returns a Promise, the callback function when the Promise succeeds takes a MediaStream object as its parameter.
6. Outliers
- Returns a Promise in a failed state, and the callback function after the Promise fails takes a DOMException object as its parameter. Possible exceptions are:
- AbortError [abort error]
- Although both the user and the operating system have granted access to the device's hardware, and there are no hardware issues that would throw a NotReadableError exception, there are still some issues that render the device unusable.
- NotAllowedError [denied error]
- The user denies the access request of the current browser instance; or the user denies the access of the current session; or the user denies all media access requests globally.
- Older versions of the specification used SecurityError, but SecurityError has been given a new meaning in newer versions.
- NotFoundError [Error not found]
- Could not find a media type that satisfies the requested parameters.
- NotReadableError[Unable to read error]
- Although the user has authorized the use of the corresponding device, a hardware, browser, or web-level error on the operating system prevents the device from being accessed.
- OverconstrainedError[Unable to meet requirements error]
- The specified requirement cannot be satisfied by the device. This exception is an object of type OverconstrainedError, which has a constraint attribute, which contains the constraint object that cannot be satisfied at present, and a message attribute, which contains a read-friendly string for Explain the situation.
- NotFoundError [Error not found]
- Could not find a media type that satisfies the requested parameters.
- NotReadableError[Unable to read error]
- Although the user has authorized the use of the corresponding device, a hardware, browser, or web-level error on the operating system prevents the device from being accessed.
- OverconstrainedError[Unable to meet requirements error]
- The specified requirement cannot be satisfied by the device. This exception is an object of type OverconstrainedError, which has a constraint attribute, which contains the constraint object that cannot be satisfied at present, and a message attribute, which contains a read-friendly string for Explain the situation.
- SecurityError [Security Error]
- Using device media is disabled on the Document on which getUserMedia() is called. Whether this mechanism is enabled or disabled depends on individual user preferences.
- TypeError [type error]
- The constraints object is not set [null], or both are set to false.