[Android]VirtualDisplay creation process and principle

Android VirtualDisplay creation process and principle

Android DisplayManager provides the createVirtualDisplay interface for creating virtual screens. The virtual screen can be used to record the screen (many information on the Internet talks about this function), split the screen (For example, a very long screen, separate different areas through the virtual screen) etc.

CreateVirtualDisplay

The function prototype in DisplayManager is as follows. The last two Hide APIs can only be used by platform applications.

// frameworks/base/core/java/android/hardware/display/DisplayManager.java
public VirtualDisplay createVirtualDisplay(@NonNull String name,
		int width, int height, int densityDpi, @Nullable Surface surface, int flags) {
    
    
}

public VirtualDisplay createVirtualDisplay(@NonNull String name,
		int width, int height, int densityDpi, @Nullable Surface surface, int flags,
		@Nullable VirtualDisplay.Callback callback, @Nullable Handler handler) {
    
    
}
/** @hide */
public VirtualDisplay createVirtualDisplay(@Nullable MediaProjection projection,
		@NonNull String name, int width, int height, int densityDpi, @Nullable Surface surface,
		int flags, @Nullable VirtualDisplay.Callback callback, @Nullable Handler handler,
		@Nullable String uniqueId) {
    
    
}
/** @hide */
public VirtualDisplay createVirtualDisplay(@Nullable MediaProjection projection,
		@NonNull VirtualDisplayConfig virtualDisplayConfig,
		@Nullable VirtualDisplay.Callback callback, @Nullable Handler handler) {
    
    
}

In addition, MediaProjection also provides the createVirtualDisplay interface, which is actually a function implemented by calling DisplayManager.


// frameworks/base/media/java/android/media/projection/MediaProjection.java
    public VirtualDisplay createVirtualDisplay(@NonNull VirtualDisplayConfig virtualDisplayConfig,
            @Nullable VirtualDisplay.Callback callback, @Nullable Handler handler) {
    
    
        DisplayManager dm = mContext.getSystemService(DisplayManager.class);
        // 调用DisplayManager的接口
        return dm.createVirtualDisplay(this, virtualDisplayConfig, callback, handler);
    }

When creating VirtualDisplay, you need to pass in Surface. **The content to be drawn on VirtualDisplay is actually displayed through the incoming Surface. **For example, create a SurfaceView on the main screen (allocate logical Display according to the physical screen), and pass this SurfaceView to VirtualDisplay. Then the content of VirtualDisplay is actually displayed on the SurfaceView on the main screen. Below is a native Android example.

// frameworks/base/packages/SystemUI/tests/src/com/android/systemui/navigationbar/NavigationBarButtonTest.java
private Display createVirtualDisplay() {
    
    
	final String displayName = "NavVirtualDisplay";
	final DisplayInfo displayInfo = new DisplayInfo();
	mContext.getDisplay().getDisplayInfo(displayInfo);

	final DisplayManager displayManager = mContext.getSystemService(DisplayManager.class);

	// 创建ImageReader，通过它得到一张Surface
	mReader = ImageReader.newInstance(displayInfo.logicalWidth,
			displayInfo.logicalHeight, PixelFormat.RGBA_8888, 2);

	assertNotNull("ImageReader must not be null", mReader);

	// 创建虚拟屏，传入Surface。
	mVirtualDisplay = displayManager.createVirtualDisplay(displayName, displayInfo.logicalWidth,
			displayInfo.logicalHeight, displayInfo.logicalDensityDpi, mReader.getSurface(),
			0 /*flags*/);

	assertNotNull("virtual display must not be null", mVirtualDisplay);

	waitForDisplayReady(mVirtualDisplay.getDisplay().getDisplayId());

	return mVirtualDisplay.getDisplay();
}

In the above example, a virtual screen is created and a Display (actually VirtualDislay) object is returned. With the Display object, we can bind the View to this virtual Display (there are many binding methods online, you can search by yourself). There are many ways to create a Surface, such as SurfaceContron+Buffer.
When VituralDisplay is created, flag needs to be provided. Its value is defined as follows, and the flags can be combined by "or".

 
    public static final int VIRTUAL_DISPLAY_FLAG_PUBLIC = 1 << 0;

    public static final int VIRTUAL_DISPLAY_FLAG_PRESENTATION = 1 << 1;

    public static final int VIRTUAL_DISPLAY_FLAG_SECURE = 1 << 2;

    public static final int VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY = 1 << 3;

    public static final int VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR = 1 << 4;

    public static final int VIRTUAL_DISPLAY_FLAG_CAN_SHOW_WITH_INSECURE_KEYGUARD = 1 << 5;

    public static final int VIRTUAL_DISPLAY_FLAG_SUPPORTS_TOUCH = 1 << 6;

    public static final int VIRTUAL_DISPLAY_FLAG_ROTATES_WITH_CONTENT = 1 << 7;

    public static final int VIRTUAL_DISPLAY_FLAG_DESTROY_CONTENT_ON_REMOVAL = 1 << 8;

    public static final int VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS = 1 << 9;

    public static final int VIRTUAL_DISPLAY_FLAG_TRUSTED = 1 << 10;

    public static final int VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP = 1 << 11;

Among the interfaces exposed by DisplayManager, there is VirtualDisplay.Callback, which provides callbacks for its status.

    public static abstract class Callback {
    
    
        /**
         * Called when the virtual display video projection has been
         * paused by the system or when the surface has been detached
         * by the application by calling setSurface(null).
         * The surface will not receive any more buffers while paused.
         */
         public void onPaused() {
    
     }

        /**
         * Called when the virtual display video projection has been
         * resumed after having been paused.
         */
         public void onResumed() {
    
     }

        /**
         * Called when the virtual display video projection has been
         * stopped by the system.  It will no longer receive frames
         * and it will never be resumed.  It is still the responsibility
         * of the application to release() the virtual display.
         */
        public void onStopped() {
    
     }
    }

VirtualDisplay principle

Regarding the implementation principle of VirtualDisplay, it is mainly analyzed from the perspective of Android Framework.

// /frameworks/base/core/java/android/hardware/display/DisplayManager.java
public VirtualDisplay createVirtualDisplay(@NonNull String name,
		int width, int height, int densityDpi, @Nullable Surface surface, int flags) {
    
    
	return createVirtualDisplay(name, width, height, densityDpi, surface, flags, null, null);
}

public VirtualDisplay createVirtualDisplay(@NonNull String name,
		int width, int height, int densityDpi, @Nullable Surface surface, int flags,
		@Nullable VirtualDisplay.Callback callback, @Nullable Handler handler) {
    
    
	final VirtualDisplayConfig.Builder builder = new VirtualDisplayConfig.Builder(name, width,
			height, densityDpi);
	builder.setFlags(flags);
	if (surface != null) {
    
    
		builder.setSurface(surface);
	}
	return createVirtualDisplay(null /* projection */, builder.build(), callback, handler);
}

// TODO : Remove this hidden API after remove all callers. (Refer to MultiDisplayService)
/** @hide */
public VirtualDisplay createVirtualDisplay(@Nullable MediaProjection projection,
		@NonNull String name, int width, int height, int densityDpi, @Nullable Surface surface,
		int flags, @Nullable VirtualDisplay.Callback callback, @Nullable Handler handler,
		@Nullable String uniqueId) {
    
    
	final VirtualDisplayConfig.Builder builder = new VirtualDisplayConfig.Builder(name, width,
			height, densityDpi);
	builder.setFlags(flags);
	if (uniqueId != null) {
    
    
		builder.setUniqueId(uniqueId);
	}
	if (surface != null) {
    
    
		builder.setSurface(surface);
	}
	return createVirtualDisplay(projection, builder.build(), callback, handler);
}

/** @hide */
public VirtualDisplay createVirtualDisplay(@Nullable MediaProjection projection,
		@NonNull VirtualDisplayConfig virtualDisplayConfig,
		@Nullable VirtualDisplay.Callback callback, @Nullable Handler handler) {
    
    
	// 走的这里，会调用到DisplayManagerGlobal中。
	return mGlobal.createVirtualDisplay(mContext, projection, virtualDisplayConfig, callback,
			handler);
}

DisplayManagerGlobal calls the DMS (DisplayManagerService) service to create a virtual screen. After getting the DisplayID returned by DMS, it creates a VirtualDisplay object on the client side through the DisplayID.

// /frameworks/base/core/java/android/hardware/display/DisplayManager.java
public VirtualDisplay createVirtualDisplay(@NonNull Context context, MediaProjection projection,
		@NonNull VirtualDisplayConfig virtualDisplayConfig, VirtualDisplay.Callback callback,
		Handler handler) {
    
    
	VirtualDisplayCallback callbackWrapper = new VirtualDisplayCallback(callback, handler);
	// 从MediaProjection过来的调用，这个地方非空。
	IMediaProjection projectionToken = projection != null ? projection.getProjection() : null;
	int displayId;
	try {
    
    
		// 告知DMS创建虚拟屏，并返回DisplayID
		displayId = mDm.createVirtualDisplay(virtualDisplayConfig, callbackWrapper,
				projectionToken, context.getPackageName());
	} catch (RemoteException ex) {
    
    
		throw ex.rethrowFromSystemServer();
	}
	if (displayId < 0) {
    
    
		Log.e(TAG, "Could not create virtual display: " + virtualDisplayConfig.getName());
		return null;
	}
	
	// 通过DisplayID，取得Display对象信息（也是调用DMS得到的）
	Display display = getRealDisplay(displayId);
	if (display == null) {
    
    
		Log.wtf(TAG, "Could not obtain display info for newly created "
				+ "virtual display: " + virtualDisplayConfig.getName());
		try {
    
    
			// 创建失败，需要释放
			mDm.releaseVirtualDisplay(callbackWrapper);
		} catch (RemoteException ex) {
    
    
			throw ex.rethrowFromSystemServer();
		}
		return null;
	}
	
	// 创建VirtualDisplay
	return new VirtualDisplay(this, display, callbackWrapper,
			virtualDisplayConfig.getSurface());
}

Create DisplayDevice in DisplayManagerService (DMS) and add it to the Device list for management

// /frameworks/base/services/core/java/com/android/server/display/DisplayManagerService.java
@Override // Binder call
public int createVirtualDisplay(VirtualDisplayConfig virtualDisplayConfig,
		IVirtualDisplayCallback callback, IMediaProjection projection, String packageName) {
    
    
		
	// 检查uid与包名，是否相符。
	final int callingUid = Binder.getCallingUid();
	if (!validatePackageName(callingUid, packageName)) {
    
    
		throw new SecurityException("packageName must match the calling uid");
	}
	if (callback == null) {
    
    
		throw new IllegalArgumentException("appToken must not be null");
	}
	if (virtualDisplayConfig == null) {
    
    
		throw new IllegalArgumentException("virtualDisplayConfig must not be null");
	}
	//  拿到client端传过来的surface对象
	final Surface surface = virtualDisplayConfig.getSurface();
	int flags = virtualDisplayConfig.getFlags();

	if (surface != null && surface.isSingleBuffered()) {
    
    
		throw new IllegalArgumentException("Surface can't be single-buffered");
	}

	// 下面开始针对Flag，做一些逻辑判断。
	if ((flags & VIRTUAL_DISPLAY_FLAG_PUBLIC) != 0) {
    
    
		flags |= VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR;

		// Public displays can't be allowed to show content when locked.
		if ((flags & VIRTUAL_DISPLAY_FLAG_CAN_SHOW_WITH_INSECURE_KEYGUARD) != 0) {
    
    
			throw new IllegalArgumentException(
					"Public display must not be marked as SHOW_WHEN_LOCKED_INSECURE");
		}
	}
	if ((flags & VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY) != 0) {
    
    
		flags &= ~VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR;
	}
	if ((flags & VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR) != 0) {
    
    
		flags &= ~VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP;
	}

	if (projection != null) {
    
    
		try {
    
    
			if (!getProjectionService().isValidMediaProjection(projection)) {
    
    
				throw new SecurityException("Invalid media projection");
			}
			flags = projection.applyVirtualDisplayFlags(flags);
		} catch (RemoteException e) {
    
    
			throw new SecurityException("unable to validate media projection or flags");
		}
	}

	if (callingUid != Process.SYSTEM_UID &&
			(flags & VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR) != 0) {
    
    
		if (!canProjectVideo(projection)) {
    
    
			throw new SecurityException("Requires CAPTURE_VIDEO_OUTPUT or "
					+ "CAPTURE_SECURE_VIDEO_OUTPUT permission, or an appropriate "
					+ "MediaProjection token in order to create a screen sharing virtual "
					+ "display.");
		}
	}
	if (callingUid != Process.SYSTEM_UID && (flags & VIRTUAL_DISPLAY_FLAG_SECURE) != 0) {
    
    
		if (!canProjectSecureVideo(projection)) {
    
    
			throw new SecurityException("Requires CAPTURE_SECURE_VIDEO_OUTPUT "
					+ "or an appropriate MediaProjection token to create a "
					+ "secure virtual display.");
		}
	}

	if (callingUid != Process.SYSTEM_UID && (flags & VIRTUAL_DISPLAY_FLAG_TRUSTED) != 0) {
    
    
		if (!checkCallingPermission(ADD_TRUSTED_DISPLAY, "createVirtualDisplay()")) {
    
    
			EventLog.writeEvent(0x534e4554, "162627132", callingUid,
					"Attempt to create a trusted display without holding permission!");
			throw new SecurityException("Requires ADD_TRUSTED_DISPLAY permission to "
					+ "create a trusted virtual display.");
		}
	}

	if (callingUid != Process.SYSTEM_UID
			&& (flags & VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP) != 0) {
    
    
		if (!checkCallingPermission(ADD_TRUSTED_DISPLAY, "createVirtualDisplay()")) {
    
    
			throw new SecurityException("Requires ADD_TRUSTED_DISPLAY permission to "
					+ "create a virtual display which is not in the default DisplayGroup.");
		}
	}

	if ((flags & VIRTUAL_DISPLAY_FLAG_TRUSTED) == 0) {
    
    
		flags &= ~VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS;
	}

	// Sometimes users can have sensitive information in system decoration windows. An app
	// could create a virtual display with system decorations support and read the user info
	// from the surface.
	// We should only allow adding flag VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS
	// to trusted virtual displays.
	final int trustedDisplayWithSysDecorFlag =
			(VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS
					| VIRTUAL_DISPLAY_FLAG_TRUSTED);
	if ((flags & trustedDisplayWithSysDecorFlag)
			== VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS
			&& !checkCallingPermission(INTERNAL_SYSTEM_WINDOW, "createVirtualDisplay()")) {
    
    
			throw new SecurityException("Requires INTERNAL_SYSTEM_WINDOW permission");
	}

	final long token = Binder.clearCallingIdentity();
	try {
    
    
		// 调用内部实现
		return createVirtualDisplayInternal(callback, projection, callingUid, packageName,
				surface, flags, virtualDisplayConfig);
	} finally {
    
    
		Binder.restoreCallingIdentity(token);
	}
}

// /frameworks/base/services/core/java/com/android/server/display/DisplayManagerService.java
private int createVirtualDisplayInternal(IVirtualDisplayCallback callback,
		IMediaProjection projection, int callingUid, String packageName, Surface surface,
		int flags, VirtualDisplayConfig virtualDisplayConfig) {
    
    
	synchronized (mSyncRoot) {
    
    
		if (mVirtualDisplayAdapter == null) {
    
    
			Slog.w(TAG, "Rejecting request to create private virtual display "
					+ "because the virtual display adapter is not available.");
			return -1;
		}
		// 为虚拟屏创建Device（告知surfaceflinger创建Display）
		DisplayDevice device = mVirtualDisplayAdapter.createVirtualDisplayLocked(
				callback, projection, callingUid, packageName, surface, flags,
				virtualDisplayConfig);
		if (device == null) {
    
    
			return -1;
		}

		// 发送添加Device通知，这里比较重要
		mDisplayDeviceRepo.onDisplayDeviceEvent(device,
				DisplayAdapter.DISPLAY_DEVICE_EVENT_ADDED);
		// 检查Display是否创建成功
		final LogicalDisplay display = mLogicalDisplayMapper.getDisplayLocked(device);
		if (display != null) {
    
    
			return display.getDisplayIdLocked();
		}

		// Something weird happened and the logical display was not created.
		Slog.w(TAG, "Rejecting request to create virtual display "
				+ "because the logical display was not created.");
		mVirtualDisplayAdapter.releaseVirtualDisplayLocked(callback.asBinder());
		mDisplayDeviceRepo.onDisplayDeviceEvent(device,
				DisplayAdapter.DISPLAY_DEVICE_EVENT_REMOVED);
	}
	return -1;
}

Next, DMS begins to call the interface of SurfaceFlinger to create a Display. And put the Display into its own List for management.

// /frameworks/base/services/core/java/com/android/server/display/VirtualDisplayAdapter.java
public DisplayDevice createVirtualDisplayLocked(IVirtualDisplayCallback callback,
		IMediaProjection projection, int ownerUid, String ownerPackageName, Surface surface,
		int flags, VirtualDisplayConfig virtualDisplayConfig) {
    
    
	String name = virtualDisplayConfig.getName();
	// VIRTUAL_DISPLAY_FLAG_SECURE 的用途，是判断是否为安全的Display，这个参数会告知SurfaceFlinger
	boolean secure = (flags & VIRTUAL_DISPLAY_FLAG_SECURE) != 0;
	IBinder appToken = callback.asBinder();
	// 调用SurfaceFligner创建Display（Display的type是virtual）
	IBinder displayToken = mSurfaceControlDisplayFactory.createDisplay(name, secure);
	final String baseUniqueId =
			UNIQUE_ID_PREFIX + ownerPackageName + "," + ownerUid + "," + name + ",";
	final int uniqueIndex = getNextUniqueIndex(baseUniqueId);
	String uniqueId = virtualDisplayConfig.getUniqueId();
	if (uniqueId == null) {
    
    
		uniqueId = baseUniqueId + uniqueIndex;
	} else {
    
    
		uniqueId = UNIQUE_ID_PREFIX + ownerPackageName + ":" + uniqueId;
	}
	// 通过SurfaceFligner返回的displayToken，创建Device对象
	VirtualDisplayDevice device = new VirtualDisplayDevice(displayToken, appToken,
			ownerUid, ownerPackageName, surface, flags, new Callback(callback, mHandler),
			uniqueId, uniqueIndex, virtualDisplayConfig);
	//  放到虚拟屏的List中管理。
	mVirtualDisplayDevices.put(appToken, device);

	try {
    
    
		if (projection != null) {
    
    
			projection.registerCallback(new MediaProjectionCallback(appToken));
		}
		appToken.linkToDeath(device, 0);
	} catch (RemoteException ex) {
    
    
		mVirtualDisplayDevices.remove(appToken);
		device.destroyLocked(false);
		return null;
	}

	// Return the display device without actually sending the event indicating
	// that it was added.  The caller will handle it.
	return device;
}

// /frameworks/base/services/core/java/com/android/server/display/DisplayDeviceRepository.java
//  mDisplayDeviceRepo.onDisplayDeviceEvent(device,
//				DisplayAdapter.DISPLAY_DEVICE_EVENT_ADDED); 
// 这段代码，会调用到下面的函数中。
private void handleDisplayDeviceAdded(DisplayDevice device) {
    
    
	synchronized (mSyncRoot) {
    
    
		DisplayDeviceInfo info = device.getDisplayDeviceInfoLocked();
		if (mDisplayDevices.contains(device)) {
    
    
			Slog.w(TAG, "Attempted to add already added display device: " + info);
			return;
		}
		Slog.i(TAG, "Display device added: " + info);
		device.mDebugLastLoggedDeviceInfo = info;
		// 需要是将Device（就是上面创建的虚拟屏幕Device）放入到DMS的管理list
		mDisplayDevices.add(device);
		// 通知Device添加，会调用到LogicalDisplayMappe的handleDisplayDeviceAddedLocked中。
		sendEventLocked(device, DISPLAY_DEVICE_EVENT_ADDED);
	}
}
// /frameworks/base/services/core/java/com/android/server/display/LogicalDisplayMapper.java
private void handleDisplayDeviceAddedLocked(DisplayDevice device) {
    
    
	DisplayDeviceInfo deviceInfo = device.getDisplayDeviceInfoLocked();
	// Internal Displays need to have additional initialization.
	// This initializes a default dynamic display layout for INTERNAL
	// devices, which is used as a fallback in case no static layout definitions
	// exist or cannot be loaded.
	if (deviceInfo.type == Display.TYPE_INTERNAL) {
    
    
		initializeInternalDisplayDeviceLocked(device);
	}

	// Create a logical display for the new display device
	LogicalDisplay display = createNewLogicalDisplayLocked(
			device, Layout.assignDisplayIdLocked(false /*isDefault*/));

	// 刷新布局和display配置
	applyLayoutLocked();
	updateLogicalDisplaysLocked();
}

To create a virtual screen, the Client creates a VirtualDisplay object through the DisplayID informed by Surface. Deal with DMS through DisplayID. On the DMS server side, create a virtual screen through SurfaceFlinger, get the DisplayToken of SurfaceFligner, and then use it to create a VirtualDisplayDevice + LogicalDisplay to manage the virtual screen.

How to get on the screen?

When creating a virtual screen, a Surface (such as a Buffer bound to the home screen) is passed in. The virtual screen gets the Buffer corresponding to the Surface through this Surface, draws the contents of the upper screen to this Buffer, and then submits it to the screen pipeline (SurfaceFlinger). Through SurfaceFlinger, this Buffer is finally drawn by SurfaceFlinger and displayed on the Display where the Surface is located (displayed according to the VirtualDisplay position.)

[Android]VirtualDisplay creation process and principle

Android VirtualDisplay creation process and principle

CreateVirtualDisplay

VirtualDisplay principle

How to get on the screen?

Guess you like