Preface

As this year’s epidemic has changed the shopping habits of various groups of people, e-commerce market share continues to rise, and live e-commerce, as a brand-new transaction method, is reshaping the flow of traffic. More and more consumers enter through live streaming Product page. Therefore, in order to obtain better live broadcast effects, anchors often spend more time preparing product highlights, preferential measures, and display links. Each link has a direct impact on the final transaction result. In the past, when merchants brought goods in a live broadcast room with a fixed setting, it was easy to cause aesthetic fatigue to the audience. When the audience did not see their counterparts, they often left because they were not interested. A viewer stays in high spirits, and the result may be that the number of live broadcast viewers does not increase but decreases with the introduction of the product.
Now with the help of the image segmentation technology launched by Huawei's machine learning service, it is possible to digitally replace various static and dynamic scenes in real time according to different product categories and needs, making the live broadcast become vivid and interesting with the switching of various styles and backgrounds. This technology uses semantic segmentation to segment the anchor’s portrait. For example, when introducing home furnishing products, you can immediately switch to a home-style room. When introducing outdoor sports equipment, you can also switch to the outdoors in real time. Through this innovative experience, viewers can also find themselves better. Immersive sense of substitution.

Features

Demo is based on the two technologies of image segmentation and hand key point recognition launched by Huawei's machine learning service. It develops the function of switching the background through gestures. In order to avoid misoperation, this demo is set to switch the background only when you wave your hand sharply. After loading a custom background Support forward switching (swipe to the right) and backward switching (swipe to the left). The operation method is the same as that of the mobile phone. It supports dynamic video background. At the same time, if you want to use customized gestures for background switching or other gesture effects, you can Integrate Huawei ML Kit hand key point recognition for customized development.

Insert picture description here
Is it a very imaginative interactive experience? Let us see how it is achieved.

Development steps

Add HUAWEI agcp plugin and Maven code base.

buildscript {
    repositories {
        google()
        jcenter()
        maven {url 'https://developer.huawei.com/repo/'}
    }
    dependencies {
        ...
        classpath 'com.huawei.agconnect:agcp:1.4.1.300'
    }
}

allprojects {
    repositories {
        google()
        jcenter()
        maven {url 'https://developer.huawei.com/repo/'}
    }
}

Full SDK mode integration.

dependencies{
     // 引入图像分割基础SDK
    implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.4.300'
    // 引入多类别分割模型包
    implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-multiclass-model:2.0.4.300'
    // 引入人像分割模型包
    implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.4.300'
    // 引入手势识别基础SDK
    implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
    // 引入手部关键点检测模型包
    implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}

Add configuration to the file header.
Add apply plugin:'com.huawei.agconnect' after apply plugin:'com.android.application'

Automatically update the machine learning model
in the AndroidManifest.xml file

<manifest
...
<meta-data
    android:name="com.huawei.hms.ml.DEPENDENCY"
    android:value="imgseg,handkeypoint" />
...
</manifest>

Create an image segmentation detector.

MLImageSegmentationAnalyzer imageSegmentationAnalyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer();//图像分割分析器
MLHandKeypointAnalyzer handKeypointAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer();//手势识别分析器

MLCompositeAnalyzer analyzer = new MLCompositeAnalyzer.Creator()
                                    .add(imageSegmentationAnalyzer)
                                   .add(handKeypointAnalyzer)
                                   .create();

Create a recognition result processing class.

public class ImageSegmentAnalyzerTransactor implements MLAnalyzer.MLTransactor<MLImageSegmentation> {
    @Override
    public void transactResult(MLAnalyzer.Result<MLImageSegmentation> results) {
        SparseArray<MLImageSegmentation> items = results.getAnalyseList();
        // 开发者根据需要处理识别结果，需要注意，这里只对检测结果进行处理。
        // 不可调用ML Kit提供的其他检测相关接口。
    }
    @Override
    public void destroy() {
        // 检测结束回调方法，用于释放资源等。
    }
}

public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
    @Override
    public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
        SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
        // 开发者根据需要处理识别结果，需要注意，这里只对检测结果进行处理。
        // 不可调用ML Kit提供的其他检测相关接口。
    }
    @Override
    public void destroy() {
        // 检测结束回调方法，用于释放资源等。
    }
}

Set the recognition result processor to realize the binding of the analyzer and the result processor.

imageSegmentationAnalyzer.setTransactor(new ImageSegmentAnalyzerTransactor());
handKeypointAnalyzer.setTransactor(new HandKeypointTransactor());

Create LensEngine

Context context = this.getApplicationContext();
LensEngine lensEngine = new LensEngine.Creator(context,analyzer)
    // 设置摄像头前后置模式，LensEngine.BACK_LENS为后置，LensEngine.FRONT_LENS为前置。
    .setLensType(LensEngine.FRONT_LENS)
    .applyDisplayDimension(1280, 720)
    .applyFps(20.0f)
    .enableAutomaticFocus(true)
    .create();

Start the camera, read the video stream, and recognize it.

// 请自行实现SurfaceView控件的其他逻辑。
SurfaceView mSurfaceView = new SurfaceView(this);
try {
    lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
    // 异常处理逻辑。
}

After the detection is complete, stop the analyzer and release the detection resources.

if (analyzer != null) {
    try {
        analyzer.stop();
    } catch (IOException e) {
        // 异常处理。
    }
}
if (lensEngine != null) {
    lensEngine.release();
}

to sum up

In summary, this little black technology can be quickly realized through a few simple steps such as introducing a package, establishing a test, analysis and result processing. In addition, through image segmentation technology, we can also do a lot of things, such as masked bullet screens in video websites, combined with some front-end rendering technologies to easily avoid bullet screens covering parts of the human body, or use existing materials to make exquisite photo photos of various sizes. One of the great benefits of semantic segmentation is that you can accurately control the objects you want to segment. In addition to portraits, you can also segment food, pets, buildings, landscapes, and even flowers and plants, and you don’t need to use professional image editing software on your computer. Up.

Github Demo

For more detailed development guidelines, please refer to the official website of Huawei Developer Alliance:https://developer.huawei.com/consumer/cn/hms/huawei-mlkit

Original link:https://developer.huawei.com/consumer/cn/forum/topic/0204395267288570031?fid=18
author: timer

Gestures switch backgrounds, making live broadcasts more immersive

Preface

Features

Development steps

to sum up

Github Demo

Guess you like