Android side to achieve multi-person audio and video chat application (2): multi-person video call

Author: Shengwang user, senior Android engineer Wu Dongyang
This series of articles shares the practical experience of implementing multi-person video calls based on Agora SDK 2.1.

In the previous article " Development of Android Multiplayer Video Chat Application (1) One-to-One Chat ", we learned how to use Agora SDK for one-to-one chat. This article mainly discusses how to use Agora SDK for multi-person chat . Mainly need to implement the following functions:
1. The chat function that has been implemented in the previous article
2. With the change of the number of participants and the resolution of their mobile phone cameras, different UIs are displayed, the so-called "split screen"
3. Click on the split screen The small window in the chat window can be enlarged and displayed

Split Screen

According to the previous technical research, the best way to display split screen is to use waterfall flow combined with dynamic chat window, which is more convenient to adapt to UI changes. The so-called waterfall flow is a relatively popular list layout at present, which will present a jagged multi-column layout on the interface. Let's first implement a waterfall flow:

There are many ways to implement waterfall flow. This article uses RecyclerView combined with GridLayoutManager to implement it. We first customize a RecyclerView named GridVideoViewContainer. The core code is as follows:

int count = uids.size();
if (count <= 2) { 
    // 只有本地视频或聊天室内只有另外一个人
    this.setLayoutManager(new LinearLayoutManager(activity.getApplicationContext(), orientation, false));
} else if (count > 2) {
    // 多人聊天室
    int itemSpanCount = getNearestSqrt(count);
    this.setLayoutManager(new GridLayoutManager(activity.getApplicationContext(), itemSpanCount, orientation, false));
}

According to the above code, it can be seen that when there is only one local video or only one other person in the chat room, LinearLayoutManager is used. This layout is actually similar to the one-to-one chat in the previous article; while in the real multi-person chat room GridLayoutManager is used to implement waterfall flow, where itemSpanCount is the column number of waterfall flow.

With an available waterfall stream, we can implement the dynamic chat window:
the point of the dynamic chat window is that the size of the item is determined by the aspect ratio of the video, so the Adapter and its corresponding layout should be careful not to write the dead size . The code to control the specific size of the item in the Adapter is as follows:

if (force || mItemWidth == 0 || mItemHeight == 0) {
    WindowManager windowManager = (WindowManager) mContext.getSystemService(Context.WINDOW_SERVICE);
    DisplayMetrics outMetrics = new DisplayMetrics();
    windowManager.getDefaultDisplay().getMetrics(outMetrics);

    int count = uids.size();
    int DividerX = 1;
    int DividerY = 1;

    if (count == 2) {
        DividerY = 2;
    } else if (count >= 3) {
        DividerX = getNearestSqrt(count);
        DividerY = (int) Math.ceil(count * 1.f / DividerX);
    }

    int width = outMetrics.widthPixels;
    int height = outMetrics.heightPixels;

    if (width > height) {
        mItemWidth = width / DividerY;
        mItemHeight = height / DividerX;
    } else {
        mItemWidth = width / DividerX;
        mItemHeight = height / DividerY;
    }
}

The above code determines the number of columns and rows based on the number of videos, then determines the width of the video based on the number of columns and screen width, and then determines the video height based on the aspect ratio and width of the video. At the same time, the horizontal and vertical screen of the mobile phone is also considered (that is, if (width > height)this line of code).

The code of the layout corresponding to the Adapter is as follows:

<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/user_control_mask"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <ImageView
        android:id="@+id/default_avatar"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true"
        android:visibility="gone"
        android:src="@drawable/icon_default_avatar"
        android:contentDescription="DEFAULT_AVATAR" />

    <ImageView
        android:id="@+id/indicator"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerHorizontal="true"
        android:layout_alignParentBottom="true"
        android:layout_marginBottom="@dimen/video_indicator_bottom_margin"
        android:contentDescription="VIDEO_INDICATOR" />

    <LinearLayout
        android:id="@+id/video_info_container"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_alignParentTop="true"
        android:layout_marginTop="24dp"
        android:layout_marginStart="15dp"
        android:layout_marginLeft="15dp"
        android:visibility="gone"
        android:orientation="vertical">

        <TextView
            android:id="@+id/video_info_metadata"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:singleLine="true"
            style="@style/NotificationUIText" />
    </LinearLayout>

</RelativeLayout>

We can see that the attributes related to the size in the layout are all wrap_content, which makes it possible for the item size to change with the video aspect ratio.

After writing the split screen layout, we can play chat video on each item.

Play chat video

In the Agora SDK, the display of a remote video is only related to the user's UID, so the data source used only needs to be simply defined as including the UID and the corresponding SurfaceView, like this:

 private final HashMap<Integer, SurfaceView> mUidsList = new HashMap<>();
 ```
每当有人加入了我们的聊天频道，都会触发`onFirstRemoteVideoDecoded(int uid, int width, int height, int elapsed)`方法，第一个 uid 就是他们的 UID；接下来我们要为每个 item 新建一个 SurfaceView 并为其创建渲染视图，最后将它们加入刚才创建好的mUidsList里并调用`setupRemoteVideo( VideoCanvas remote )`方法播放这个聊天视频。这个过程的完整代码如下:

 ```
@Override
public void onFirstRemoteVideoDecoded(int uid, int width, int height, int elapsed) {
    doRenderRemoteUi(uid);
}

private void doRenderRemoteUi(final int uid) {
    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            if (isFinishing()) {
                return;
            }

            if (mUidsList.containsKey(uid)) {
                return;
            }

            SurfaceView surfaceV = RtcEngine.CreateRendererView(getApplicationContext());
            mUidsList.put(uid, surfaceV);

            boolean useDefaultLayout = mLayoutType == LAYOUT_TYPE_DEFAULT;

            surfaceV.setZOrderOnTop(true);
            surfaceV.setZOrderMediaOverlay(true);

            rtcEngine().setupRemoteVideo(new VideoCanvas(surfaceV, VideoCanvas.RENDER_MODE_HIDDEN, uid));

            if (useDefaultLayout) {
                log.debug("doRenderRemoteUi LAYOUT_TYPE_DEFAULT " + (uid & 0xFFFFFFFFL));
                switchToDefaultVideoView();
            } else {
                int bigBgUid = mSmallVideoViewAdapter == null ? uid : mSmallVideoViewAdapter.getExceptedUid();
                log.debug("doRenderRemoteUi LAYOUT_TYPE_SMALL " + (uid & 0xFFFFFFFFL) + " " + (bigBgUid & 0xFFFFFFFFL));
                switchToSmallVideoView(bigBgUid);
            }
        }
    });
}

The above code is the same as the code for playing one-to-one video in the previous article, but careful readers may have found that we did not put the generated SurfaceView in the interface, which is the difference from one-to-one video: we To release SurfaceView in an abstract VideoViewAdapter class, the key code is as follows:

SurfaceView target = user.mView;
VideoViewAdapterUtil.stripView(target);
holderView.addView(target, 0, new FrameLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT));

Generally, Android engineers will understand that this is the root layout of ViewHolder's layout when they see holderView, and where does the user come from, see the code at the end of the article, and I won't go into details in the article.

In this way, when multiple people are chatting, we can use the split-screen method to play user chat videos. What if we want to zoom in on a user's video?

Full screen and small windows

When the user double-clicks an item, he wants the corresponding video to be displayed in full screen, while other videos become small windows, then we first define a double-click event interface:

public interface VideoViewEventListener {
    void onItemDoubleClick(View v, Object item);
}
具体实现方式如下：
mGridVideoViewContainer.setItemEventHandler(new VideoViewEventListener() {
    @Override
    public void onItemDoubleClick(View v, Object item) {
        log.debug("onItemDoubleClick " + v + " " + item + " " + mLayoutType);

        if (mUidsList.size() < 2) {
            return;
        }

        UserStatusData user = (UserStatusData) item;
        int uid = (user.mUid == 0) ? config().mUid : user.mUid;

        if (mLayoutType == LAYOUT_TYPE_DEFAULT && mUidsList.size() != 1) {
            switchToSmallVideoView(uid);
        } else {
            switchToDefaultVideoView();
        }
    }
});

The method of playing the selected video in full screen is easy to understand, we only look at the method of generating the small window list:

private void switchToSmallVideoView(int bigBgUid) {
    HashMap<Integer, SurfaceView> slice = new HashMap<>(1);
    slice.put(bigBgUid, mUidsList.get(bigBgUid));
    Iterator<SurfaceView> iterator = mUidsList.values().iterator();
    while (iterator.hasNext()) {
        SurfaceView s = iterator.next();
        s.setZOrderOnTop(true);
        s.setZOrderMediaOverlay(true);
    }

    mUidsList.get(bigBgUid).setZOrderOnTop(false);
    mUidsList.get(bigBgUid).setZOrderMediaOverlay(false);

    mGridVideoViewContainer.initViewContainer(this, bigBgUid, slice, mIsLandscape);

    bindToSmallVideoView(bigBgUid);

    mLayoutType = LAYOUT_TYPE_SMALL;

    requestRemoteStreamType(mUidsList.size());
}

The list of small windows should pay attention to the UID that removes the full screen. In addition, everything is the same as the normal waterfall view, including double-clicking the item of the small window to play it in full screen.

At this point, we have used the Agora SDK to complete a simple multi-person chat demo with basic functions. There is still a lot of things to be done to be commercialized. Let’s make a brief summary here!

Summarize

Agora provides a high-quality video communication SDK, which not only covers the mainstream operating systems, but also has high integration efficiency, and also supports multiple modes of video calls including chat, conference, live broadcast and other functions. The API design in the SDK can basically meet most of the development needs, and hides the underlying development. You only need to provide SurfaceView and UID to play videos, which is very friendly to developers at the App layer. It is very suitable for developers with video chat development needs. Today, with the explosion of entrepreneurship in the video field, it is recommended that more developers who want to engage in this field can try it.