This article is exclusively authorized by Jianxi. Jianxi is also doing Android multimedia development. He is a fellow. However, he mainly focuses on video compression. Using FFmpeg, he can do many things, but the effect is not much. Look at his sharing today, Jianxi's blog is: Jianxi's blog_CSDN blog-audio and video, Android common sense, blogger in the field of computer basics .
1. Warm up
Time flies, time flies, it has been two or three months since the last bragging, many women around me have split and reunited, reunited and split, this dick is still proudly single. Last time, we roughly talked about some simple FFmpeg commands and the simple calling method of the Java layer, and then many friends left me messages on github or csdn. Many times I chose to avoid answering, because the library used to use The so package is not open source, and I can't change the contents at all. But this time, let's play something big. I recompiled FFmpeg and rewritten the interface functions of JNI. This time, I will fully open source from C to Java. The 2.0 project took more than two months of my spare time, and it was finally completed today. It's very jelly, and this blog will express all the author's heartfelt thoughts, whether it's very jelly or not. In addition to chicken jelly, I also want to complain about the efficiency of its soft stools. It is really not very high. I will try hard coding in 3.0, or I will use H265 coding in the 2.0 iteration. This is something later. But looking at the rhythm of WeChat replacing small videos with big ones, it should be possible.
This article involves knowledge points:
Andorid video and audio capture
YUV video processing (manual cutting, rotation, mirroring, etc.) PCM audio processing
Using FFmpeg API, YUV encoding is H264, PCM encoding is AAC
Configuration of FFmpeg encoder
The Practical Application of JNI in Engineering
Production and application of FFmpeg command tool under Android
The application of Android Studio plug-in cMake in the project
Charge:
At least you need to know what YUV, PCM, and MP4 are (zero-based learning method for video and audio codec technology).
It is best to read and compile FFmpeg available under Android (including libx264 and libfdk-aac), compile FFmpeg with executable commands under Android, and play JNI under Android. Most of the knowledge will not be repeated.
Have a basic understanding of C/C++ basic syntax.
personal environment and tools
System: macOS-10.12.5
Compiler: Android Studio-2.3.2
ndk: r14
FFmpeg: 3.2.5
★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.
see below!
2. Project overview:
2.1 Effect drawing:
The project address has not changed: GitHub - mabeijianxi/small-video-record: use FFmpeg video to record WeChat small video and its compression processing. Here, the gif image of version 1.0 is reused, because the interface has not changed at all, and the functions are not packaged so much for the time being, it does not matter. It will be added later.
2.2 Overall process:
2.3 Project catalog browsing:
3. New project
We create a new project, which may be different from the past. When we need to check the C++ support and C++ standard options, select C++ 11, as shown below:
C++ support is a must, and there is a reason for choosing C++11. We will use some APIs in it later. Then we compiled the six dynamic libraries, header files and cmdutils.c cmdutils.h cmdutils_common_opts.h config.h ffmpeg.c ffmpeg.h ffmpeg_filter compiled in FFmpeg (including libx264 and libfdk-aac) available under Android .c ffmpeg_opt.c copy to the cpp directory of our project. After completion, your cpp directory should be as follows
Maybe you will have one more automatically generated native-lib.cpp than me, this file will keep it temporarily.
4. Write the JNI interface:
I created a new interface class FFmpegBridge.java, and temporarily defined the following methods according to my needs:
package com.mabeijianxi.smallvideorecord2.jniinterface;import java.util.ArrayList;/** * Created by jianxi on 2017/5/12. * https://github.com/mabeijianxi * [email protected] */public class FFmpegBridge { private static ArrayList<FFmpegStateListener> listeners=new ArrayList(); static { System.loadLibrary("avutil"); System.loadLibrary("swresample"); System.loadLibrary("avcodec"); System.loadLibrary("avformat"); System.loadLibrary("swscale"); System.loadLibrary("avfilter"); System.loadLibrary("jx_ffmpeg_jni"); * End the recording and save the transcoding } /** */ public static final int ALL_RECORD_END =1; public final static int ROTATE_0_CROP_LF=0; /** * rotate 90 degrees to crop top left */ public final static int ROTATE_90_CROP_LT =1; /** * not processed yet */ public final static int ROTATE_180= 2; /** * Rotate 270 (-90) to crop the upper left, left and right mirror */ public final static int ROTATE_270_CROP_LT_MIRROR_LR=3; /** * * @return return ffmpeg compilation information */ public static native String getFFmpegConfig(); / ** * Run ffmpeg in command form * @param cmd * @return returns 0 to indicate success */ private static native int jxCMDRun(String cmd[]); /** * Encode one frame of video, only yv12 video can be encoded temporarily * @param data * @return */ public static native int encodeFrame2H264(byte[] data); /** * Encode one frame of audio, temporarily only encode pcm audio * @ param data * @return */ public static native int encodeFrame2AAC(byte[] data); /** * end of recording * @return */ public static native int recordEnd(); /** * initialization * @param debug * @param logUrl */ public static native void initJXFFmpeg(boolean debug,String logUrl); public static native void nativeRelease(); /** * * @param mediaBasePath video storage directory * @param mediaName video name * @param filter rotation mirror clipping processing * @param in_width input video width * @param in_height input video height * @param out_height output video height * @param out_width output video width * @param frameRate video frame rate * @param bit_rate video bit rate * @ return */ public static native int prepareJXFFmpegEncoder(String mediaBasePath, String mediaName, int filter, int in_width, int in_height, int out_width, int out_height, int frameRate, long bit_rate); /** * command form execution* @param cmd * / public static int jxFFmpegCMDRun(String cmd){ String regulation="[ \\t]+"; final String[] split = cmd.split(regulation); return jxCMDRun(split); } /** * Underlying callback * @param state * @param what */ public static synchronized void notifyState(int state,float what){ for(FFmpegStateListener listener: listeners){ if(listener!=null){ if(state== ALL_RECORD_END){ listener.allRecordEnd(); } } } } /** *注册录制回调 * @param listener */ public static void registFFmpegStateListener(FFmpegStateListener listener){ if(!listeners.contains(listener)){ listeners.add(listener); } } public static void unRegistFFmpegStateListener(FFmpegStateListener listener){ if(listeners.contains(listener)){ listeners.remove(listener); } } public interface FFmpegStateListener { void allRecordEnd(); } }Copy code
When you create these methods, because the native is not defined, they will all become popular at this time, don’t worry and don’t get entangled, put the cursor on the corresponding method, and press Alt + Enter lightly, and the effect as shown in the figure will appear:
After confirming again, this interface will be added in native. I don't like calling native-lib.cpp very much, so I changed it to jx_ffmpeg_jni.cpp
5. Write native code
I don't use c/c++ much, and I'm used to Java, so sometimes I struggle with naming, what should I do if I don't like it? Then bear with it a little bit~~
5.1 Prepare the log function:
No matter what language you play, you can play wool without a log, so this is the first step. Create new jx_log.cpp and jx_log.h. jx_log.h:
/** * Created by jianxi on 2017/6/2. * https://github.com/mabeijianxi * [email protected] */#ifndef JIANXIFFMPEG_JX_LOG_H#define JIANXIFFMPEG_JX_LOG_H#include <android/log.h>extern int JNI_DEBUG;#define LOGE(debug, format, ...) if(debug){__android_log_print(ANDROID_LOG_ERROR, "jianxi_ffmpeg", format, ##__VA_ARGS__);}#define LOGI(debug, format, ...) if(debug){__android_log_print(ANDROID_LOG_INFO, "jianxi_ffmpeg", format, ##__VA_ARGS__);}#endif //JIANXIFFMPEG_JX_LOG_H复制代码
jx_log.cpp:
/** * Created by jianxi on 2017/6/2. * https://github.com/mabeijianxi * [email protected] */#include "jx_log.h"int JNI_DEBUG= 1; copy code
Of course, we also define a flag JNI_DEBUG whether to enable debug.
5.2 Prepare the FFmpeg interface of the executable command:
It is assumed here that you have finished reading FFmpeg that compiles executable commands under Android, because we need to make some changes to the source code that was copied in before, otherwise it will not work. We create two new files to connect to FFmpeg. One function in the file is called for the Java layer, one is called for Native, and the other is used for initializing the debug control log, which can be left alone.
jx_ffmpeg_cmd_run.h：
/** * Created by jianxi on 2017/6/4. * https://github.com/mabeijianxi * [email protected] */#ifndef JIANXIFFMPEG_FFMPEG_RUN_H#define JIANXIFFMPEG_FFMPEG_RUN_H#include <jni.h>JNIEXPORT jint JNICALLJava_com_mabeijianxi_smallvideorecord2_jniinterface_FFmpegBridge_jxCMDRun(JNIEnv *env, jclass type, jobjectArray commands);void log_callback(void* ptr, int level, const char* fmt, va_list vl);JNIEXPORT void JNICALLJava_com_mabeijianxi_smallvideorecord2_jniinterface_FFmpegBridge_initJXFFmpeg(JNIEnv *env, jclass type, jboolean debug, jstring logUrl_);int ffmpeg_cmd_run(int argc, char **argv);#endif //JIANXIFFMPEG_FFMPEG_RUN_H复制代码
jx_ffmpeg_cmd_run.c:
/** * Created by jianxi on 2017/6/4.. * https://github.com/mabeijianxi * [email protected] */#include "jx_ffmpeg_cmd_run.h"#include "ffmpeg.h"#include "jx_log.h"/** * 以命令行方式运行，返回0表示成功 */JNIEXPORT jint JNICALLJava_com_mabeijianxi_smallvideorecord2_jniinterface_FFmpegBridge_jxCMDRun(JNIEnv *env, jclass type, jobjectArray commands){ int argc = (*env)->GetArrayLength(env,commands); char *argv[argc]; int i; for (i = 0; i < argc; i++) { jstring js = (jstring) (*env)->GetObjectArrayElement(env,commands, i); argv[i] = (char *) (*env)->GetStringUTFChars(env,js, 0); } return ffmpeg_cmd_run(argc,argv); }int ffmpeg_cmd_run(int argc, char **argv){ return jxRun(argc, argv); }char *logUrl;/** * 初始化debug工具 */JNIEXPORT void JNICALLJava_com_mabeijianxi_smallvideorecord2_jniinterface_FFmpegBridge_initJXFFmpeg(JNIEnv *env, jclass type, jboolean debug, jstring logUrl_) { JNI_DEBUG = debug; if (JNI_DEBUG&&logUrl_!=NULL) { av_log_set_callback(log_callback); const char* log = (*env)->GetStringUTFChars(env,logUrl_, 0); logUrl = (char*)malloc(strlen(log)); strcpy(logUrl,log); (*env)->ReleaseStringUTFChars(env,logUrl_, log); } }void log_callback(void *ptr, int level, const char *fmt, va_list vl) { FILE *fp = NULL; if (!fp) fp = fopen(logUrl, "a+"); if (fp) { vfprintf(fp, fmt, vl); fflush(fp); fclose(fp); } } Copy code
If you write this in one breath, it will definitely become popular everywhere. It will be terrible. You can’t find any files or methods. That’s because you have added so many files. The cMake tool doesn’t know. The correct way is to add a C/ C++ file and then go to CMakeLists.txt to tell others, don't forget to click Sync to synchronize when you are done.
5.3 Prepare a secure queue:
After we collect audio and video data, we will send it to FFmpeg for a series of processing. Because it is soft coding, the speed of coding has a lot to do with the CPU. The current x264 algorithm combined with today’s CPU cannot keep up with our collection At a speed of 20 frames per second +, if you directly capture a frame and encode a frame, you will definitely lose the frame, so I decided to put it in a queue. Due to the existence of multi-threaded programming, our queue needs safety, just a few It's like a man snatching a girl, the girl naturally needs someone like me to protect her. The code of this queue is copied from my website, so there is nothing to say~~
threadsafe_queue.cpp
/** * Created by jianxi on 2017/5/31. * https://github.com/mabeijianxi * [email protected] */#ifndef JIANXIFFMPEG_THREADSAFE_QUEUE_CPP#define JIANXIFFMPEG_THREADSAFE_QUEUE_CPP#include <queue>#include <memory>#include <mutex>#include <condition_variable>/** * 一个安全的队列 */template<typename T>class threadsafe_queue {private: mutable std::mutex mut; std::queue<T> data_queue; std::condition_variable data_cond;public: threadsafe_queue() {} threadsafe_queue(threadsafe_queue const &other) { std::lock_guard<std::mutex> lk(other.mut); data_queue = other.data_queue; } void push(T new_value)//入队操作 { std::lock_guard<std::mutex> lk(mut); data_queue.push(new_value); data_cond.notify_one(); } void wait_and_pop(T &value)//until there is an element that can be deleted { std::unique_lock<std::mutex> lk(mut); data_cond.wait(lk, [this] { return !data_queue.empty(); }); value = data_queue.front(); data_queue.pop(); } std::shared_ptr<T> wait_and_pop() { std::unique_lock<std::mutex> lk(mut); data_cond.wait(lk, [this] { return !data_queue.empty(); }); std::shared_ptr<T> res(std::make_shared<T>(data_queue.front())); data_queue.pop(); return res; } bool try_pop(T &value)//不管有没有队首元素直接返回 { std::lock_guard<std::mutex> lk(mut); if (data_queue.empty()) return false; value = data_queue.front(); data_queue.pop(); return true; } std::shared_ptr<T> try_pop() { std::lock_guard<std::mutex> lk(mut); if (data_queue.empty()) return std::shared_ptr<T>(); std::shared_ptr<T> res(std::make_shared<T>(data_queue.front())); data_queue.pop(); return res; } bool empty() const { return data_queue.empty(); } };#endif //JIANXIFFMPEG_THREADSAFE_QUEUE_CPP复制代码
The several libs used here are in the C++11 standard~
5.4 Prepare a structure for storing configuration information:
In fact, this thing is similar to JavaBean, directly engage in code, the JXJNIHandler field in the code is not seen for the time being.
jx_user_arguments.h:
/** * Created by jianxi on 2017/5/26. * https://github.com/mabeijianxi * [email protected] */#ifndef JIANXIFFMPEG_JX_USER_ARGUMENTS_H#define JIANXIFFMPEG_JX_USER_ARGUMENTS_H#include "jni.h"class JXJNIHandstr UserArguments { const char *media_base_path; //File storage address const char *media_name; //File command prefix char *video_path; //Video storage address char *audio_path; //Audio storage address char *media_path; //Synthesized MP4 Storage address int in_width; //output width int in_height; //input height int out_height; //output height int out_width; //output width int frame_rate; //video frame rate control long long video_bit_rate; //video bit rate control int audio_bit_rate; //Audio bit rate control int audio_sample_rate; //audio sample rate control (44100) int v_custom_format; //some filters Operation control JNIEnv *env; //env global pointer JavaVM *javaVM; //jvm pointer jclass java_class; //calss object of java interface class JXJNIHandler *handler; // pointer to a global processing object} ;#endif //JIANXIFFMPEG_JX_USER_ARGUMENTS_H
This structure will be used throughout the process.
5.5 Write video (YUV) encoding code
This section is one of the cores of this article. The simplified idea is as follows:
Some brothers may ask why not encode a frame to synthesize a frame, because I tested the synthesis time, and it is basically at the millisecond level, and it is too troublesome. If I do this, I will directly use the FFmpeg command tool we made and then Just a few lines of code
The code is pasted, now let me talk about its past and present life, it is very important~.
1) Video encoder parameter configuration
Here are a few important ones. If you haven’t complained about the parameters for a while, you can open it here and take a closer look at the configuration parameters of the ffmpeg encoder AVCodecContext.
size_t path_length = strlen(arguments->video_path); char *out_file = (char *) malloc(path_length + 1); strcpy(out_file, arguments->video_path);复制代码
Through the above code, we copied the video output address. It is very important that our video output address ends with .h264, because the following avformat_alloc_output_context2(&pFormatCtx, NULL, NULL, out_file) function will check its legality, and according to your suffix format Corresponds to pFormatCtx assignment.
-
pCodecCtx->codec_id = AV_CODEC_ID_H264 The encoder id is specified here, which is undoubtedly H264;
-
pCodecCtx->pix_fmt = AV_PIX_FMT_YUV420P; Specify the encoded data format;
-
pCodecCtx->bit_rate = arguments->video_bit_rate, specifies the video bit rate, this parameter is very important, largely determines the quality and size of your video, but according to this is also related to the bit rate mode in VBR mode, it will have a certain fluctuations.
-
pCodecCtx->thread_count = 16 The number of threads, I have written it to death here, it is not very good, friends on the road said that 1.5 is enough to accompany the number of cores.
-
pCodecCtx->time_base.num = 1; pCodecCtx->time_base.den = arguments->frame_rate These two control the frame rate, num is the denominator, den is the numerator, and the frame rate is obtained by dividing. You must be the same as the frame rate you collected. You are very important here, otherwise it may cause the video and audio to be out of sync, and you will pass by. The frame number you set for your camera is not necessarily the actual frame number saved. At this time It will also cause the video and audio to be out of sync, which will be discussed later when it is connected to the Java layer.
-
av_opt_set(pCodecCtx->priv_data, "preset", "superfast", 0) Here is a preset value for specifying a coding speed, which I hardcoded as the fastest for the time being.
-
pCodecCtx->qmin pCodecCtx->qmax This is the quantization range setting, its value ranges from 0 to 51, the smaller the value, the higher the quality, and the greater the required bit rate, 0 means lossless encoding. About the coding process and principle, you can read the basic principles of video compression coding and audio compression coding
-
pCodecCtx->max_b_frames = 3 The maximum b frame is 3, which can be set to 0 so that the encoding will be faster, because the motion estimation and motion compensation encoding time is divided into i, b, p frames, borrow a word from Raytheon: I frame only uses this frame It does not require motion estimation and motion compensation during the encoding process. Obviously, since the I frame does not eliminate the correlation in the time direction, the compression ratio is relatively low. In the encoding process of the P frame, a previous I frame or P frame is used as a reference image for motion compensation, which actually encodes the difference between the current image and the reference image. The encoding method of a B frame is similar to that of a P frame, the only difference is that it uses a previous I frame or P frame and a subsequent I frame or P frame for prediction during the encoding process. It can be seen that the encoding of each P frame needs to use one frame of images as a reference image, while the B frame requires two frames of images as a reference. In contrast, B frames have a higher compression ratio than P frames, so there will be a certain delay if there are more B frames.
-
av_dict_set(¶m, "profile", "baseline", 0) It can limit your output to a specific H.264 profile, all profiles include: baseline, main.high, high10, high422, high444, pay attention to use -- The profile option is incompatible with lossless encoding.
2) The YUV data structure collected by the Android camera
Let’s briefly talk about the YUV format first. Similar to RGB, YUV is also a color coding method. Y: represents brightness (Luminance or Luma), that is, grayscale values; while U and V: represent chroma (Chrominance or Chroma ), the function is to describe the color and saturation of the image, and is used to specify the color of the pixel. If there is only Y, then it is black and white audio and video. According to different sampling methods, there are mainly YUV4:4:4, YUV4:2:2, and YUV4:2:0. Its YUV 4:4:4 sampling, each Y corresponds to a set of UV components. YUV 4:2:2 sampling, every two Y share a set of UV components. YUV 4:2:0 sampling, every four Y shares a set of UV components. For example, if there are eight pixels on the screen, YUV4:4:4 will have 8 Y, 8 U, and 8 V. YUV4:2:2 will have 8 Y, 4 U, and 4 V. YUV4:2:0 will have 8 Y, 2 U, and 2 V. If we want to process the data we collect, we must know its data type and data structure. In the old version of android sdk, it can only collect data in two modes, YV12 and NV12. They both belong to YUV420, but the other The arrangement structure is different. Let’s take a look at the picture below. Of course, I’ve photoshopped the first picture below, because the original picture is wrong, but I’m old and my hands are not perfect, so I just read it.
It can be seen that the four physically similar pixels Y1, Y2, Y7, and Y8 share the same U1 and V1, and the similar Y3, Y4, Y9, and Y10 use U2 and V2. The different colors here portray this feature very vividly, which is clear at a glance. The number of grids is the size of the byte array of this frame of image, and the arrangement order of the array elements is the same as the long bar behind. NV12 is as follows:
It can be found that they are only different in the emission position of UV.
3) YV12 data processing
It is possible to use both YV12 and NV12. I chose YV12 when configuring the camera parameters. Next, we write a few simple algorithms to realize the cutting and rotation of the video. It is very simple. I estimated that it would be like this at the time. up.
Here we assume that the video we collected is 640 wide and 480 high, and we want to cut it into a video with a width of 400 and a height of 300. According to the above knowledge, we can specify that there will be 640 480 Y in a frame of byte array of 640 480 , and it will be at the top, then there will be (1/4) 640 480 V, and then there will be (1/4) 640 480 A U, we want to cut it into 400 300, naturally we just need to keep part of the data. We first build a model for Y, since it is 640 480, we can regard it as a row with 640 Y, a total of 480 lines, as shown in the figure below, the red mark indicates 640*480 Y, and the yellow area is our Clip all values of Y done.
Note the orientation of the image. With this model we can write code to manipulate arrays. Here is a piece of code:
Cut Y:
unsigned char *in_buf; unsigned char *out_buf_y; for(int i=480-300;i<480;i++){//Traverse high
for(int j=0;j<400;j++){//Traverse wide
int index =640*i+j;//Unsigned char value of the subscript currently traversed
=*(in_buf+index);//Y value under the current subscript//Start assigning to our target array
*(out_buf_y+(i- (480-300))*400+j)=value;//The target array is 400*300, here it is traversed and assigned in order from the 0 corner mark} }
copy
code
Assuming that in_buf is a frame of YV12 video data, after executing this cycle, we will get the cut Y value. Next, we analyze the cut UV data. The UV model is a little different from Y. The reason why it is called YUV4: 2: 0 is not because there is no V, it is actually UV exchange scanning in the vertical direction, such as scanning U in the first line, scanning V in the second line, and scanning U in the third line. In the horizontal direction, it is scanned every other time. For example, if the first column is scanned, the second column will not be scanned, and then the third column will be scanned. Therefore, the data of U in the horizontal and vertical directions is 1/2 of its Y, and the total number is 1/4 of its value, and the same is true for V. Knowing these we can easily build a model.
The area of 320-240 is the area where we are our U value or V value, and the area of 200-150 is the target area of our cut U value or V value. code show as below:
Cut UVs:
unsigned char *in_buf; unsigned char *out_buf_u; unsigned char *out_buf_v; for(int i=(480-300)/2;i<480/2;i++){//traversal high for(int j=0;j
< 400/2;j++){//traversal width
int
index=(640/2)*i+j;//unsigned char
v=*(in_buf+(640*480)+index);/ /The V value under the current corner mark (the pointer position has to be moved back 640*480 units first, because Y is placed in front)
unsigned
char u=*(in_buf+(640*480*5/4)+index); //The U value under the current subscript (the pointer position must be moved backward by 640*480*5/4 units, because Y and V are placed in front)//Assigned to our target array out_buf_u from the subscript 0
*(out_buf_u+(i-(480-300)/2)*400/2+j)=u;
*(out_buf_v+(i-(480-300)/2)*400/2+j)=v;
}
} copy code
After the above operations, we have completed the most basic cutting. The data collected by the camera is horizontal. If we record vertically and we do not do any operations, then the recorded video is rotated 90° counterclockwise. Tnd, if you go counterclockwise, the brother will turn you 90° clockwise, so it should be straight.
With the idea, as shown in the above figure, our for loop remains unchanged, because the position that needs to be cut remains unchanged, we only change the discharge position of the output array, the original first row is placed in the last column, and the second row is placed in the penultimate Two columns, inferred from this. The following also demonstrates with code:
Y shear and rotate 90° clockwise:
unsigned char *in_buf; unsigned char *out_buf_y; for(int i=(480-300);i<480;i++){//traverse height
for(int j=0;j<400;j++){//traverse width
int
index=(640)*i+j;//Unsigned char
value
=*(in_buf+index);//The Y value under the current corner mark
*
(out_buf_y+j*300+ (300-(i-(480-300)-1)))=value;//Combine the image of the output array to understand
}
}copy code
It is very simple when Y is done with UV, because we have already mastered the law, the value of UV in both horizontal and vertical directions is half of Y.
Cut UVs:
unsigned char *in_buf; unsigned char *out_buf_u; unsigned char *out_buf_v; for(int i=(480-300)/2;i<480/2;i++){//Traverse high for(int j=0;j
< 400/2;j++){//traversal width
int
index=(640/2)*i+j;//unsigned char
value_v
=*(in_buf+(640*480)+index); //V value under the current corner mark
unsigned char value_u=*(in_buf+(640*480*5/4)+index);//U value under the current corner mark
*
(out_buf_u+j*300/2+( 300/2-(i-(480-300)/2-1)))=value_u;//Combine the image of the output array to understand
*(out_buf_v+j*300/2+(300/2-(i- (480-300)/2-1)))=value_v;//Combine the image of the output array to understand
}
}copy code
Because of the front camera, it will cause mirror image, so when recording with the front camera, you need to process the mirror image. For more details, please refer to the source code. In addition to these, we can do many interesting operations, such as when the UV value is assigned to 128 When it becomes a black bar image, you can also adjust the brightness and tone and so on.
After processing the data, call the API of FFmpeg encoding.
5.6 Audio Encoding
From the flow chart above, we can see that the steps are similar to those of the video, and the amount of data is relatively small. If you use libfdk-aac to compile, you can basically catch up with the collection speed. Serve first, then chat:
jx_pcm_encode_aac.h:
/** * Created by jianxi on 2017/5/18. * https://github.com/mabeijianxi * [email protected] */#ifndef JIANXIFFMPEG_JX_PCM_ENCODE_AAC_H#define JIANXIFFMPEG_JX_PCM_ENCODE_AAC_H#include "base_include.h"#include "jx_user_arguments.h"using namespace std;/** * pcm编码为aac */class JXPCMEncodeAAC {public: JXPCMEncodeAAC(UserArguments* arg);public: int initAudioEncoder(); static void* startEncode(void* obj); void user_end(); int sendOneFrame(uint8_t* buf); int encodeEnd();private: int flush_encoder(AVFormatContext *fmt_ctx, unsigned int stream_index);private: threadsafe_queue<uint8_t *> frame_queue; AVFormatContext *pFormatCtx; AVOutputFormat *fmt; AVStream *audio_st; AVCodecContext *pCodecCtx; AVCodec *pCodec; AVFrame *pFrame; AVPacket pkt; int got_frame = 0; int ret = 0; int size = 0; int i; int is_end=0; UserArguments *arguments; ~JXPCMEncodeAAC() { } };#endif //JIANXIFFMPEG_JX_PCM_ENCODE_AAC_H
I don't do so much research on audio. The following is just a brief introduction to the parameters. More accessible video and audio data processing entry: PCM audio sampling data processing
Encoding parameters:
-
pCodecCtx->sample_fmt = AV_SAMPLE_FMT_S16 Set its sampling format, ours is a 16-bit unsigned integer, which needs to correspond to the parameters set when Java audio is collected.
-
pCodecCtx->sample_rate = arguments->audio_sample_rate Sampling rate, audio is not the most important thing for us, here I wrote the mainstream 44100, here also need to correspond to the parameters set when Java audio collection.
-
pCodecCtx->channel_layout = AV_CH_LAYOUT_MONO; pCodecCtx->channels = av_get_channel_layout_nb_channels(pCodecCtx->channel_layout) This is to set the number of channels. Since the audio requirements are not high, I use a single channel, which also needs to correspond to the parameters set during Java audio acquisition. . There are many options such as AV_CH_LAYOUT_STEREO is a stereo two-channel, AV_CH_LAYOUT_4POINT0 is a 4-channel.
-
pCodecCtx->bit_rate = arguments->audio_bit_rate Audio bit rate.
After configuring the parameters, the rest will be handed over to FFmpeg.
5.7 Write video synthesis class
After the audio and video are encoded, we need to synthesize them into mp4. Now we can use the FFmpeg command tool we have prepared. We only need to throw the address to it. This synthesis process takes very little time.
jx_media_muxer.h:
/** * Created by jianxi on 2017/5/24. * https://github.com/mabeijianxi * [email protected] */#ifndef JIANXIFFMPEG_JX_MEDIA_MUXER_H#define JIANXIFFMPEG_JX_MEDIA_MUXER_H#include "base_include.h"class JXMediaMuxer{public: int startMuxer(const char * video, const char *audio , const char *out_file);private: };#endif //JIANXIFFMPEG_JX_MEDIA_MUXER_H
Original link: Use FFmpeg to play Android video recording and compression - Nuggets
★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.
see below!