Under LAN, and one-click screen sharing to mobile devices

Under LAN, and one-click screen sharing to mobile devices

1. The cause of the problem

---

Development needs

Soon a newly developed education class app, teachers need to implement end-to-student mobile devices for remote control, such as student tablet unlock screen, the computer screen to sharing student end, monitoring student screen content.

Web environment

Teachers end of the cable or WIFI access, iPad and Android Pad WIFI access by ensuring that, in a segment.

Roughly function

graph TB S (Service <br/> teacher terminal) S - Canton a key to unlock / lock the screen -> <br/> C1 S-- a key distribution files ppt / doc / img -> C2 S-- screen broadcast -> C3 S-- student Responder -> C4 S-- time monitoring -> C5 C1 (Client1 <br/> iPad / Android Pad) C2 (Client2 <br/> iPad / Android Pad) C3 (Client3 < br /> iPad / Android Pad) C4 (Client4 <br/> iPad / Android Pad) C5 (Client4 <br/> iPad / Android Pad)

2. implementation

---

Teachers end uses FFmpeg acquisition screen audio and video, iOS, Android end use ijkplayer pull streaming, streaming protocol uses RTMP protocol, communication using TCP the Socket .

Communication in

After the teacher to act as an LAN server sends a UDP broadcast (content includes the local IP and port number), iOS, Android client receives UDP broadcasts the acquired IP address and port, using Socket [ CocoaAsyncSocket (the iOS), Socket (the Android)] and after the teacher terminal TCP connection, the connection establishment is completed, send and receive messages through the Socket communication.

3. Technical Module

Mac set up under nginx-full

---

1> Homebrew Installation

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2> installation nginx-full (rtmp)

brew install nginx-full --with-rtmp-module

3> path etc. See installation information nginx

brew info nginx-full

It will show the path configuration file

The default port has been set in /usr/local/etc/nginx/nginx.conf to 8080 so that
nginx can run without sudo.

4> nginx.conf configuration, the file was last added directly blank (application live, live just named, after plug-flow corresponding to it).

rtmp {
    server {
        listen 1935;
        application live {
            live on;
            record off;
        }
    }
}

After 5> Modify saved restart nginx

nginx -s reload

Mounting plug flow ffmepg

---

1> Install

brew install ffmpeg

2> Push the screen flow

ffmpeg -f avfoundation -pixel_format uyvy422 -i "1" -f flv rtmp://localhost:1935/live

Display Output address, rtmp after execution: // localhost: 1935 / live, which is the machine ip, such as rtmp: //192.168.1.2: 1935 / live, Mac computers can install the VLC player to play test.

Output #0, flv, to 'rtmp://localhost:1935/live':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: flv1 (flv) ([2][0][0][0] / 0x0002), yuv420p, 2560x1600, q=2-31, 200 kb/s, 1000k fps, 1k tbn, 1000k tbc
    Metadata:
      encoder         : Lavc58.35.100 flv
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
frame=  241 fps= 27 q=24.8 size=    5368kB time=00:00:08.86 bitrate=4958.5kbits/s speed=   1x     

IJKPlayer compilation

---

Accessories: after iOS and Android compiled DLL libraries

Reference ijkplayer documented in mac to compile, but before compiling, you need to modify some configuration files. If you want to reach the first seconds on screen, be sure to read these contents go to compile, including mentioned later clients seconds on the first screen after, because it involves the C file modification, but also eliminates the need to recompile.

Be sure to read README.md before compiling, such as compiling the environment and the required documents:

# install homebrew, git, yasm
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install git
brew install yasm

# add these lines to your ~/.bash_profile or ~/.profile
# export ANDROID_SDK=<your sdk path>
# export ANDROID_NDK=<your ndk path>

# on Cygwin (unmaintained)
# install git, make, yasm

There is, he was the compiler environment My Build Environment, this need to explain, particularly to compile Andrews, NDK directly with r10e, although you can also later, but there will likely fail to compile, because I compile time failed, replaced by the version used by the author.

Common
Mac OS X 10.11.5
Android
NDK r10e
Android Studio 2.1.3
Gradle 2.14.1
iOS
Xcode 7.3 (7D175)
HomeBrew
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install git

README.md correspondence There Build iOS and Build Android, which platform compiler to execute the corresponding command. Where the default link script is less codec / format for smaller binary size, specific instructions to see the document, I chose the default configuration.

Build iOS compilation, ./init-ios.shcommand a little longer in the middle to download something, you can view the details of the script file. For example == pull ffmpeg base ==, significantly for a long time, unless you were to download very quickly.

== pull ffmpeg base ==
Cloning into 'extra/ffmpeg'...
remote: Enumerating objects: 538907, done.
Receiving objects:  19% (103984/538907), 30.82 MiB | 42.00 KiB/s   

iOS Framwork merger

---

After successful completion of all running demo, get compile a dynamic library, here I direct use of real machine simulator and combined dynamic library, of course, you can not merge, direct use of real machine dynamic libraries.

1. Release mode configuration, Edit Scheme -> Run -> info -> Build Configuration -> Release
2. Real machine and compile their own simulator
3. Products —> IJKMediaFramework.framework —> Show in Finder
4. A terminal cd Products directory, execute: lipo -create 真机 模拟器 -output 合并文件.

lipo -create Release-iphoneos/IJKMediaFramework.framework/IJKMediaFramework Release-iphonesimulator/IJKMediaFramework.framework/IJKMediaFramework -output IJKMediaFramework

5. Replace the merged files off the file under real machine framework, new IJKMediaFramework.framework is combined dynamic library, drag and drop directly into the project.

iOS compiler may encounter problems and solutions

Question 1:

./libavutil/arm/asm.S:50:9: error: unknown directive
        .arch armv7-a
        ^
make: *** [libavcodec/arm/aacpsdsp_neon.o] Error 1
make: *** Waiting for unfinished jobs....

Solution:

Modify ./compile-ffmpeg.sh file

This line: FF_ALL_ARCHS_IOS8_SDK = "armv7 arm64 i386 x86_64"

Amended to: FF_ALL_ARCHS_IOS8_SDK = "arm64 i386 x86_64"

Question 2:

'openssl/ssl.h' file not found
#include <openssl/ssl.h> ERROR: openssl not found

Solution:

Soft compile ffmpeg codec library, this process generates various architectures ffmpeg, ffmpeg compiled before the first compile OpenSSL, to compile openssl, if not executed can cause an error. You must perform ./compile-openssl.sh all

The actual compiler does encounter these problems, especially the problem 1.

My Xcode Version Version 10.3 (10G8)

Accessories: after iOS and Android compiled DLL libraries

These questions refer to the blog iOS IJKPlayer integrated project

Client seconds on the first screen

---

The first seconds on screen, requires a combination of clarity and video delay, take appropriate frame rate. The client can also reduce the cache cancel the first keyframe display time. With specific reference to the fold seconds on the frame and chase playback technology .

Attach iOS and Android to IJKPlayer settings.

iOS end:

- (IJKFFOptions *)options {
    if (!_options) {
        IJKFFOptions *options = [IJKFFOptions optionsByDefault];
        // Set param
        [options setFormatOptionIntValue:1024 * 16 forKey:@"probsize"];
        [options setFormatOptionIntValue:50000 forKey:@"analyzeduration"];
        [options setPlayerOptionIntValue:0 forKey:@"videotoolbox"];
        [options setCodecOptionIntValue:IJK_AVDISCARD_DEFAULT forKey:@"skip_loop_filter"];
        [options setCodecOptionIntValue:IJK_AVDISCARD_DEFAULT forKey:@"skip_frame"];
        [options setPlayerOptionIntValue:1000 forKey:@"max_cached_duration"];
        [options setPlayerOptionIntValue:1 forKey:@"infbuf"];  // 无限读
        [options setPlayerOptionIntValue:0 forKey:@"packet-buffering"];
        _options = options;
    }
    return _options;
}

Android side:

// 设置播放前的最大探测时间
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "analyzemaxduration", 100L);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "probesize", 10240L);
// 每处理一个packet之后刷新io上下文
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "flush_packets", 1L);
// 是否开启预缓冲，一般直播项目会开启，达到秒开的效果，不过带来了播放丢帧卡顿的体验
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "packet-buffering", 0L);
// 放前的探测Size，默认是1M, 改小一点会出画面更快
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "probsize", 200);
// 设置播放前的探测时间 1,达到首屏秒开效果
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "analyzeduration", 1);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "max_cached_duration", 1000);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "infbuf", 1);
// 无限读
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "max-buffer-size", 0);
// 不额外优化（使能非规范兼容优化，默认值0 ）
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "fast", 1);
// 缩短播放的rtmp视频延迟在1s内
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "fflags", "nobuffer");
// 如果是rtsp协议，可以优先用tcp(默认是用udp)
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "rtmp_transport", "tcp");
// 支持硬解 1：开启 O:关闭
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "mediacodec-hevc", 1);
// 跳帧处理,放CPU处理较慢时，进行跳帧处理，保证播放流程，画面和声音同步
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "rtsp_transport", "tcp");
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "framedrop", 1L);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_PLAYER, "start-on-prepared", 1);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "http-detect-range-support", 0);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_CODEC, "skip_loop_filter", 48L);
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_CODEC, "skip_frame", 0);
// 因为项目中多次调用播放器，有网络视频，resp，本地视频，还有wifi上http视频，所以得清空DNS才能播放WIFI上的视频
ijkMediaPlayer.setOption(IjkMediaPlayer.OPT_CATEGORY_FORMAT, "dns_cache_clear", 1);

Modify f_ffplay.c compiled before, which significantly improves the fold delays

Path ijkmedia-> ijkplayer -> ff_ffplay.c

The first modification places: double vp_duration method

This code

static double vp_duration(VideoState *is, Frame *vp, Frame *nextvp) {
    if (vp->serial == nextvp->serial) {
        double duration = nextvp->pts - vp->pts;
        if (isnan(duration) || duration <= 0 || duration > is->max_frame_duration)
            return vp->duration;
        else
            return duration;
    } else {
        return 0.0;
    }
}

Replace the code

static double vp_duration(VideoState *is, Frame *vp, Frame *nextvp) {
     return vp->duration;
}

The second modification of the place: static int ffplay_video_thread (void * arg) method

Comment out AVRational frame_rate = av_guess_frame_rate(is->ic, is->video_st, NULL);this line of code

will duration = (frame_rate.num && frame_rate.den ? av_q2d((AVRational){frame_rate.den, frame_rate.num}) : 0);

change intoduration = 0.01;

static int ffplay_video_thread(void *arg)
{
    FFPlayer *ffp = arg;
    VideoState *is = ffp->is;
    AVFrame *frame = av_frame_alloc();
    double pts;
    double duration;
    int ret;
    AVRational tb = is->video_st->time_base;
  	// 注释掉
    // AVRational frame_rate = av_guess_frame_rate(is->ic, is->video_st, NULL);
    int64_t dst_pts = -1;
    int64_t last_dst_pts = -1;
    int retry_convert_image = 0;
    int convert_frame_count = 0;
  
    // ···此处省略很多代码
  	
#endif
						// 这行代码直接修改为 duration = 0.01;
						// duration = (frame_rate.num && frame_rate.den ? av_q2d((AVRational){frame_rate.den, frame_rate.num}) : 0);
            duration = 0.01;
            pts = (frame->pts == AV_NOPTS_VALUE) ? NAN : frame->pts * av_q2d(tb);
            ret = queue_picture(ffp, frame, pts, duration, frame->pkt_pos, is->viddec.pkt_serial);
            av_frame_unref(frame);
#if CONFIG_AVFILTER
        }
#endif

        if (ret < 0)
            goto the_end;
    }
 the_end:
#if CONFIG_AVFILTER
    avfilter_graph_free(&graph);
#endif
    av_log(NULL, AV_LOG_INFO, "convert image convert_frame_count = %d\n", convert_frame_count);
    av_frame_free(&frame);
    return 0;
}

Modify f_ffplay.c reference to the blog ijkplayer optimized recording of some of the problems