Overview
This article will introduce how to use ffmpeg + canvas to write an online visual audio and video editing tool:
- How to run ffmpeg in the browser to process audio and video;
- How to use canvas to implement a video track;
- How to combine tracks, players and ffmpeg to achieve video editing visualization;
Key words
ffmpeg, video, track, visualization, canvas
Effect screenshot
Experience address
videoCut(deployed in vercel, requires scientific Internet access)
code repository
img-generate (I have clicked in, please click a free star for me)
Background of the project
Reasons for making the project
In work and life, we often encounter some audio and video related problems, such as:
- Want to capture a picture from a video;
- Want to capture a specific segment from a long video;
- Format conversion, extract audio from video;
- Get video information such as frame rate;
If you analyze locally, you must first configure the environment and install analysis tools such as ffmpeg, which requires more preparation work. If you use wasm technology and run ffmpeg directly on the browser, there is no need for local installation and downloading, which will be more friendly to novice users.
Try it yourself - using ffmpeg in the browser
@ffmpeg/ffmpeg
This is a compiled package. The disadvantage is that it is relatively large and you need to wait a few seconds when loading for the first time; the advantage is that you can use it directly. Here we directly reference this package, and when optimization is needed later, we will consider compiling it ourselves or looking for a more suitable package.
api introduction
Just look at an official demo and with a few lines of script, you can complete a format conversion task.
import { writeFile } from 'fs/promises';
import { createFFmpeg, fetchFile } from '@ffmpeg/ffmpeg';
// 设置参数
const ffmpeg = createFFmpeg({ log: true });
(async () => {
// 加载 ffmpeg.wasm-core 脚本
await ffmpeg.load();
// 输入文件需要调用 FS 方法，这样 ffmpeg 才能够进行”消费“
ffmpeg.FS('writeFile', 'test.avi', await fetchFile('./test.avi'));
// 执行 ffmpeg 命令
await ffmpeg.run('-i', 'test.avi', 'test.mp4');
// node 端将生成的文件直接写到文件中
await fs.promises.writeFile('./test.mp4', ffmpeg.FS('readFile', 'test.mp4'));
process.exit(0);
})();The approximate process is as follows
Initialize project
I am using umi here, which can be initialized through the following command
# 新建文件夹
mkdir myapp && cd myapp
# 初始化项目
yarn create @umijs/umi-app
# 安装依赖
yarn startAfter completing the above steps, you can see the page effect on the browser.
Introduce @ffmpeg/ffmpeg
The installation package is very simple, just pass npm i @ffmpeg/ffmpeg , but during the introduction process, I encountered some problems
Question 1: Module parse failed: Unexpected token
webpack4
ERROR in ./node_modules/@ffmpeg/ffmpeg/src/browser/defaultOptions.js 7:68
Module parse failed: Unexpected token (7:68)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| */
| const corePath = typeof process !== 'undefined' && process.env.NODE_ENV === 'development'
> ? new URL('/node_modules/@ffmpeg/core/dist/ffmpeg-core.js', import.meta.url).href
| : `https://unpkg.com/@ffmpeg/core@${devDependencies['@ffmpeg/core'].substring(1)}/dist/ffmpeg-core.js`;
|
@ ./node_modules/@ffmpeg/ffmpeg/src/browser/index.js 1:23-50
@ ./node_modules/@ffmpeg/ffmpeg/src/index.js
@ ./src/pages/index.tsx
@ ./src/.umi/core/routes.ts
@ ./src/.umi/umi.ts
@ multi ./node_modules/umi/node_modules/@umijs/preset-built-in/bundled/@pmmmwh/react-refresh-webpack-plugin/client/ReactRefreshEntry.js ./src/.umi/umi.tsSolution 1: Change the import method
import { createFFmpeg, fetchFile } from '@ffmpeg/ffmpeg';Change to
import { createFFmpeg, fetchFile } from '@ffmpeg/ffmpeg/dist/ffmpeg.min.js';At this time, the type will be lost. Just set it forcibly.
import { FFmpeg } from '@ffmpeg/ffmpeg';
const ffmpeg = createFFmpeg({
...
}) as FFmpeg;Solution 2: Switch to webpack 5
umi switching webpack 5 is relatively simple, just configure it directly in .umirc
webpack5: {}Then place the core file under public and set the parameters when creatingFFmpeg.
const ffmpeg = createFFmpeg({
...
corePath: `${location.origin}/static/v0.11.0/ffmpeg-core.js`,
});Question 2: SharedArrayBuffer is not defined
The cause of the problem is that ffmpeg uses SharedArrayBuffer, but SharedArrayBuffer has some restrictions due to security issues. If you want to use SharedArrayBuffer in the page, you need to set the page for cross-domain isolation; or set the token token as a temporary solution. (The specific reasonThe front-end FFmpeg? may not be ready yetThe introduction is more detailed, you can refer to it)
Solution: Set up cross-domain isolation
CORP can be set by setting two request headers in the page
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-originIn development, by setting up devServer, the problem can be solved.
devServer: {
headers: {
'Cross-Origin-Opener-Policy': 'same-origin',
'Cross-Origin-Embedder-Policy': 'require-corp',
},
},Sort out the steps and run through the basic process
After successfully introducing the @ffmpeg/ffmpeg package, you can start developing functions. According to the principle of function priority, the functions should be run through first, and the process and UI can be optimized later.
Referenced several demos浏览器 ffmpeg and compiled a more comfortable process.
Upload files to be processed
Get the file to be uploaded and then call the FS method
const props = {
...,
beforeUpload: async (file) => {
const { name, size, type } = file;
ffmpeg.FS('writeFile', name, await fetchFile(file));
...
setOpInput(name);
return false;
},
}
...
<Upload {...props}>
<Button icon={<UploadOutlined />}>选择文件</Button>
</Upload>Excuting an order
After 输入, 输出文件, 执行脚本 and other parameters are filled in, you can merge the parameters and run ffmpeg 命令, the output result is obtained.
await ffmpeg.run(...allArgs.split(' '));Take a look at the effect, it's no problem
Add a little optimization
The ffmpeg command is actually relatively difficult to remember, and it is not friendly for users to find it by themselves. So the expectation is to record all the more commonly used commands so that users can choose them directly.
export const getOp = (op: string, args?: IGetOp) => {
// const { out = OUT_DEFAULT, input = IN_DEFAULT, timer } = args || {}
let output = '';
let resultOp = '';
const { rangeLeft, rangeRight, input, out, timer } = args || {};
switch (op) {
case OP_NAME.screenshot:
resultOp = `-i ${input} -ss ${timer} -vframes 1 ${out}`;
output = 'out.png';
break;
case OP_NAME.getMp3FromVideo:
resultOp = ` -f mp3 -vn`;
output = 'out.mp3';
break;
case OP_NAME.getInfo:
resultOp = '';
output = OUT_DEFAULT;
break;
case OP_NAME.custom:
resultOp = '';
output = OUT_DEFAULT;
break;
case OP_NAME.cutVideo:
resultOp = `-ss ${rangeLeft} -to ${rangeRight} -i ${input} -c:v copy -c:a copy ${out}`;
output = OUT_DEFAULT;
break;
default:
resultOp = DEFAULT_ARGS;
output = OUT_DEFAULT;
}
return [resultOp, output];
};The effect is as shown below
Experience address
ffmpeg(deployed in vercel, requires scientific Internet access)
Video track (visual audio and video segmentation)
In order to implement可视化的音视频分割, a track and player need to be implemented. Set the segmentation points in a precise manner through the track, and see the video effect of the points in real time through the player. Here we refer to the open source project shWave to implement a simple track.
Overall analysis
If the track is split, it is divided into:
- background
- scale
- Time pointer (pointing to the moment when the current video is playing)
Then we implement it in the order of 静态部分 -> 动态部分（参数传递） -> 动态部分（事件响应、缩放条） .
Static partial implementation
Add canvas to page
Tracks are implemented using canvas, so you must first add a canvas to the page
<canvas
ref={$canvas}
id="shcanvas"
style={
{
height: '100%',
width: '100%',
zIndex: 0,
pointerEvents: 'auto',
}}
></canvas>After the canvas is loaded, the drawing of the track begins.
useEffect(() => {
if (!waveCanvas) { return }
draw?.()
}, [waveCanvas, draw, currentTime, duration]);
// 各种场景都有可能触发重新绘制drawIn the function, the three elements mentioned (background, scale, time pointer) are drawn respectively.
const draw = () => {
const ctx = waveCanvas && waveCanvas?.getContext("2d");
if (!waveCanvas || !ctx) return;
//绘制背景
drawBackground(waveCanvas, ctx, backgroundColor);
// 刻度尺
drawRuler(waveCanvas, ctx, 1, duration)
// 时间指针
drawPointer({
canvas: waveCanvas,
ctx,
pixelRatio: 1,
duration,
currentTime,
color: pointerColor,
pointerWidth,
})
}Draw the background (drawBackground)
export const drawBackground = (canvas: HTMLCanvasElement, ctx: CanvasRenderingContext2D, backgroundColor = "#529393") => {
if (!canvas || !ctx) return;
const { width, height } = canvas;
//清空上一次渲染
ctx.clearRect(0, 0, width, height);
//设置填充绘画的颜色
ctx.fillStyle = backgroundColor;
//填充出一个矩形 （绘制背景）
ctx.fillRect(0, 0, width, height);
};Show results
draw scale
The scale is to loop through, and then draw small rectangles of different lengths, and then under the whole second scale, there will be text displayed:
- Whole second, longest;
- 0.5 seconds, second;
- 0.1 seconds, the shortest;
whole second
length is required秒长度 * 10, then iterate and draw the longest rectangle every 10 * 0.1. Among them, each time it is traversed, what changes is the position of x, and other parameters remain unchanged.
for (let index = 0; index < length; index += 1) {
//十格间距
if (index % 10 === 0) {
// x , y , w, h
ctx.fillRect(
index * gap,
0,
pixelRatio,
fontHeight * pixelRatio);
}
}
0.5 seconds
Everything else remains the same, only the length has changed.
else if (index % 5 === 0) {
ctx.fillRect(index * gap, 0, pixelRatio, (fontHeight * pixelRatio) / 1.5);
}
0.1 seconds
Again, just shorter in length.
else {
ctx.fillRect(index * gap, 0, pixelRatio, (fontHeight * pixelRatio) / 3);
}
duration text display text is the moment corresponding to reality,时刻 = begin + second
if (index % 10 === 0) {
second += 1;
ctx.fillText(
// text x y maxWidth
// time = 开始时间 + 遍历到的秒数
durationToTime(begin + second).split(".")[0], // s
gap * index - fontSize * pixelRatio * 2 + pixelRatio, // x
fontTop * pixelRatio // y
);
}
Timeline (timer)
The time scale is actually a rectangle, indicating the current moment of video playback, which is the time scale corresponding to currentTime.
const { width, height } = canvas;
const length = getLength(duration);
// 每 0.1 s 所对应的像素宽度。
const gap = getGap(width, length)
// 开始点
const begin = getBegin(currentTime, duration);
ctx.fillRect(
Number(((currentTime - begin) * 10 * gap).toFixed(3)), // x
0, // y
pointerWidth * pixelRatio, // width
height, // height
)
By doing this, the effect of a static timeline is already available.
Parameter passing
Next, we pass all the variables in as parameters, so that the timeline can be animated. Here, we use storybook to view the real-time effect.
argTypes: {
currentTime: {
control: {
type: 'number',
},
},
duration: {
control: {
type: 'number',
},
},
backgroundColor: {
control: {
type: 'color',
},
},
pointerWidth: {
control: {
type: 'number',
},
},
pointerColor: {
control: {
type: 'color',
},
},
},The effect is as shown in the figure. Modify the input parameters below and you will see the real-time effect above.
incident response
zoom bar
shwave is scaled by adjusting the duration. One problem with this solution is that when zooming in, the duration at the back cannot be seen, and only the duration at the front can be zoomed in. I think a more appropriate way would be to add a scaling ratio field to control the length of the track. Add a scroll bar below. After placing it on the track, you can scroll to view the subsequent time.
Use Slider + InputNumber to implement a zoom bar that can be dragged and input.
<Col span={5}>
<Slider
min={1}
max={20}
onChange={(value: number) => setRatio(value)}
value={ratio}
step={0.1} />
</Col>
<Col span={8}>
<InputNumber
min={1}
max={20}
style={
{ margin: '0 16px' }}
step={0.1}
value={ratio}
onChange={(value: number | null) => setRatio(value || 1)}
/>
</Col>
track click
After clicking on the track, you need to calculate the corresponding time of the click, and then reset the currentTime
First, monitor the canvas
useEffect(() => {
if (waveCanvas === null) {
return
}
//设置canvas点击监听
waveCanvas.addEventListener("click", onCanavsClick);
return () => {
waveCanvas.removeEventListener("click", onCanavsClick);
}
}, [waveCanvas])After listening to the event click, directly calculate the corresponding time and set it.
const onCanavsClick = (event: MouseEvent) => {
const time = computeTimeFromEvent(event);
if (currentTime !== time) {
click?.(time);
}
};The offset moment is the calculation step:
- Calculate the pixel size occupied by each;
- Calculate the offset size of the click position from the left side;
- Based on the offset, calculate the corresponding moment in the timeline;
const computeTimeFromEvent = (event: MouseEvent) => {
if (!waveCanvas || !$shwave.current) {
return 0
}
const { clientWidth: width } = waveCanvas;// canvas 实际宽度
const pixelRatio = window.devicePixelRatio; // 1
const length = getLength(duration); // 100
const gap = getGap(width, length); // 0.1 s 所占用的像素 宽度
// 偏移的宽度
const left = event.pageX - $shwave.current.offsetLeft / pixelRatio;
const begin = getBegin(currentTime, duration);
// left 在 时间中的位置
const time = clamp(
((left / gap) * pixelRatio) / 10 + begin,
begin,
begin + duration
);
return time;
}player
After the track is completed, a video player must be implemented to see the video effect of the points in real time.
The player here is a video element. In order to realize the linkage between the track and the player, it is necessary to monitor some video events and set the currentTime during playback;
set video
First place a video element in the interface
const videoRef = useRef<HTMLVideoElement | null>(null)
<video controls width={400} height={300} src={url} ref={videoRef}></video> Then monitor, set duration in canplay event, and set currentTime in ontimeupdate.
useEffect(() => {
if (!videoRef.current) {
return
}
videoRef.current.ontimeupdate = () => {
setCurrentTime?.(videoRef.current?.currentTime)
}
videoRef.current.addEventListener('canplay', () => {
setDuration?.(videoRef.current?.duration)
})
}, [url])Basic effect display
Visual video editing
After ffmpeg runs in the browser, track and player linkage are completed, we can realize the function of visual video editing.
video segmentation
In the ffmpeg command, to implement the editing function, you need to provide the start point time and end point time.
ffmpeg -ss 00:17:24 -to 02:19:31 -i inputVideo.mp4 -c:v copy -c:a copy outputVideo.mp4So a complete process should look like this:
Effect screenshot
Video screenshot
After playing the video to the specified location, we can take a screenshot of the video by executing the ffmpeg command,
ffmpeg -i ${input} -ss ${timer} -vframes 1 ${out}Effect screenshot
Other details
Save uploaded files locally
After uploading a file, if you refresh the page, the uploaded file will be lost, so it is necessary to add a local cache. This way, after refreshing the page by mistake, you can still get the previously uploaded files. IndexDb is chosen to store files because the storage space is large enough. The more commonly used localStorage storage space is relatively small, with a maximum storage space of only 5M.
Initialization
In the react project, there are some ready-made libraries that can be used directly, the most commonly used one is dexie. We first introduce dexie to define the database table.
// db.ts
import Dexie, { Table } from 'dexie';
export interface FileData {
id?: number;
name: string;
type: string;
data: File;
}
export class FileDexie extends Dexie {
files!: Table<FileData>;
constructor() {
super('myDatabase');
this.version(1).stores({
files: '++id, name, type, data' // Primary key and indexed props
});
}
}
export const dbFileDexie = new FileDexie();Save uploaded file
After indexDb initialization is completed, we start uploading files, and then call the function in the upload component's onChange function to save the file to in indexDb. db.files.put
const handleMediaChange = (event: React.ChangeEvent<HTMLInputElement>) => {
const file = event.target.files?.[0];
if (file) {
try {
const { name } = file;
db.files.put({ name, type: file.type, data: file })
} catch (error) {
console.error('handleMediaChange error', error)
}
};
}
<input type="file" onChange={handleMediaChange} />Where used, call useLiveQuery to query.
import { useLiveQuery } from 'dexie-react-hooks';
import { dbFileDexie as db } from '@/db'
const mediaList = useLiveQuery(
() => db.files?.toArray?.()
);Production environment deployment
Since ffmpeg relies on cross-source isolation, response headers need to be configured, and deployment on github is not possible. So I found another free deployment platform vercel . The disadvantage is that domestic Internet access needs to be scientific. There are many tutorials on vercel deployment, so I won’t go into details. Just share vercel.json, which implements setting response headers and supporting history routing.
{
"rewrites": [
{ "source": "/(.*)", "destination": "/index.html" }
],
"headers": [
{
"source": "/(.*)",
"headers": [
{ "key": "Cross-Origin-Opener-Policy", "value": "same-origin" },
{ "key": "Cross-Origin-Embedder-Policy", "value": "require-corp" }
]
}
]
}ALL
To achieve this, I have actually just completed a demo. But I feel like there’s still a lot more that can be done. If you further improve this project, it can provide convenience for your future work and life and make it a useful tool.
- Style optimization
- Audio visualization, intercepting audio clips
- Test material generation
- Support subtitle function (separate, add)
- Compile ffmpeg by yourself, reduce the package size, and remove the dependency on SharedArrayBuffer.
reference
FFmpeg on the front end? Probably not ready yet
Use ffmpeg.wasm pure front-end to achieve multi-audio and video synthesis
Front-end video frame extraction ffmpeg + Webassembly
Project Reference
Original text How to use ffmpeg + canvas to write an online visual audio and video editing tool
★The business card at the end of the article allows you to receive free audio and video development learning materials, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmap, etc.
See below! ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓