[QT] Qt ApplicationManager Compositor source code analysis

Analysis of Compositor function of Qt ApplicationManager

• According to the Qt ApplicationManager official website, it implements the Compositor function based on the Wayland protocol. The following is the official website introduction. In fact, QtApplicationManager uses the QtWayland module to implement Compositor. Wayland is a set of Compositor standards designed to replace XWindow. Those who are interested can learn about it on their own.

To support multiple UI processes on an embedded Linux system, you need a central window compositor: a Wayland compositor is the state-of-the-art solution for this. Consequently, the application manager incorporates a compositor that is fully-compliant with the Wayland protocol, based on the QtWayland module.

• Regarding Wayland Compositor, simply put, the Client draws Buffers, and the Compositor merges these Buffers and finally displays them on the screen. It probably looks like the picture below (picture taken from QtWayland official website).
  
• QtApplicationManager recommends using QML for development. A set of Compositor services (processes) can be written in QML.

Here is a Compositor Sample based on QML to analyze how QtApplicationManager implements the Compositor function.

A QML Compositor example

• The Hello World program is provided under the path qtapplicationmanager\examples\applicationmanager\hello-world, in which system-ui.qml (startup entry) is implemented as follows

/****************************************************************************
**
** Copyright (C) 2021 The Qt Company Ltd.
** Copyright (C) 2019 Luxoft Sweden AB
** Copyright (C) 2018 Pelagicore AG
** Contact: https://www.qt.io/licensing/
**
** This file is part of the QtApplicationManager module of the Qt Toolkit.
**
** $QT_BEGIN_LICENSE:BSD$
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
** a written agreement between you and The Qt Company. For licensing terms
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
**
** BSD License Usage
** Alternatively, you may use this file under the terms of the BSD license
** as follows:
**
** "Redistribution and use in source and binary forms, with or without
** modification, are permitted provided that the following conditions are
** met:
**   * Redistributions of source code must retain the above copyright
**     notice, this list of conditions and the following disclaimer.
**   * Redistributions in binary form must reproduce the above copyright
**     notice, this list of conditions and the following disclaimer in
**     the documentation and/or other materials provided with the
**     distribution.
**   * Neither the name of The Qt Company Ltd nor the names of its
**     contributors may be used to endorse or promote products derived
**     from this software without specific prior written permission.
**
**
** THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
** "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
** LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
** A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
** OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
** SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
** LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
** DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
** THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
** (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
** OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
**
** $QT_END_LICENSE$
**
****************************************************************************/

import QtQuick 2.4
import QtApplicationManager.SystemUI 2.0

Item {
    // 省略
    // 重点关注这段代码
    // Show windows
    Column {
        anchors.right: parent.right
        Repeater {
            model: WindowManager
            WindowItem {
                width: 600
                height: 200
                window: model.window
            }
        }
    }
}

• In the above code, a series of window objects inside the WindowManager object are bound to WindowItem, and the width and height are set. Because WaylandSurface is one-to-one with the window object in WindowManager (will be explained later), this code is the essence of making Surface displayed.

• When creating WaylandSurface, WindowManager creates a Window object. The specific implementation is as follows.

// 绑定了QWaylandSurface::hasContentChanged信号，当有该信号时触发surfaceMapped信号。
// 该信号在Surface中有Buffer被Commit时触发
// 这块代码可以看出来 QtApplicationManager目前只支持 XDG和WlShell两种类型。

void WaylandCompositor::createWlSurface(QWaylandSurface *surface, const QWaylandResource &resource)
{
    
    
    WindowSurface *windowSurface = static_cast<WindowSurface *>(surface);
    QWaylandWlShellSurface *ss = new QWaylandWlShellSurface(m_wlShell, windowSurface, resource);
    windowSurface->setShellSurface(ss);

    connect(windowSurface, &QWaylandSurface::hasContentChanged, this, [this, windowSurface]() {
    
    
        if (windowSurface->hasContent())
            emit this->surfaceMapped(static_cast<WindowSurface*>(windowSurface));
    });
}


void WaylandCompositor::onXdgSurfaceCreated(QWaylandXdgSurface *xdgSurface)
{
    
    
    WindowSurface *windowSurface = static_cast<WindowSurface*>(xdgSurface->surface());

    Q_ASSERT(!windowSurface->m_wlSurface);
    windowSurface->m_xdgSurface = xdgSurface;

    connect(windowSurface, &QWaylandSurface::hasContentChanged, this, [this, windowSurface]() {
    
    
        if (windowSurface->hasContent())
            emit this->surfaceMapped(windowSurface);
    });
    emit windowSurface->xdgSurfaceChanged();
}

void WindowManager::registerCompositorView(QQuickWindow *view)
{
    
    
    // 省略

#if defined(AM_MULTI_PROCESS)
    if (!ApplicationManager::instance()->isSingleProcess()) {
    
    
        if (!d->waylandCompositor) {
    
    
            d->waylandCompositor = new WaylandCompositor(view, d->waylandSocketName);
            for (const auto &extraSocket : d->extraWaylandSockets)
                d->waylandCompositor->addSocketDescriptor(extraSocket);

            connect(d->waylandCompositor, &QWaylandCompositor::surfaceCreated,
                    this, &WindowManager::waylandSurfaceCreated);
            connect(d->waylandCompositor, &WaylandCompositor::surfaceMapped,
                    this, &WindowManager::waylandSurfaceMapped);

       // 省略
    }
#else
    if (!once)
        qCInfo(LogGraphics) << "WindowManager: running in single-process mode [forced at compile-time]";
#endif
    // 省略
}

// 这里函数里面，会创建Window对象，并将Window对象与Surface对象绑定。
void WindowManager::waylandSurfaceMapped(WindowSurface *surface)
{
    
    
    qint64 processId = surface->processId();
    const auto apps = ApplicationManager::instance()->fromProcessId(processId);
    Application *app = nullptr;

    if (apps.size() == 1) {
    
    
        app = apps.constFirst();
    } else if (apps.size() > 1) {
    
    
        // if there is more than one app within the same process, check the XDG surface appId
        const QString xdgAppId = surface->applicationId();
        if (!xdgAppId.isEmpty()) {
    
    
            for (const auto &checkApp : apps) {
    
    
                if (checkApp->id() == xdgAppId) {
    
    
                    app = checkApp;
                    break;
                }
            }
        }
    }

// 只有开启了非安全模式，才可以接收非App侧创建的Surface
    if (!app && ApplicationManager::instance()->securityChecksEnabled()) {
    
    
        qCCritical(LogGraphics) << "SECURITY ALERT: an unknown application with pid" << processId
                                << "tried to map a Wayland surface!";
        return;
    }

    Q_ASSERT(surface);

    qCDebug(LogGraphics) << "Mapping Wayland surface" << surface << "of" << d->applicationId(app, surface);

    // Only create a new Window if we don't have it already in the window list, as the user controls
    // whether windows are removed or not
    int index = d->findWindowByWaylandSurface(surface->surface());
    if (index == -1) {
    
    
        WaylandWindow *w = new WaylandWindow(app, surface);
        // 这里会将Window绑定到List上
        setupWindow(w);
    }
}

• Why can binding Window (corresponding to Surface) to WindowItem display the content of Surface? Assigning Window to WindowItem actually calls the following function. So in fact, it is the Surface setting that is given to QWaylandQuickItem.

void WindowItem::WaylandImpl::setup(Window *window)
{
    
    
    Q_ASSERT(!m_waylandWindow);
    Q_ASSERT(window && !window->isInProcess());

    m_waylandWindow = static_cast<WaylandWindow*>(window);

    if (!m_waylandItem)
        createWaylandItem();

    m_waylandItem->setBufferLocked(false);
    m_waylandItem->setSurface(m_waylandWindow->surface());
}

void WindowItem::WaylandImpl::createWaylandItem()
{
    
    
    m_waylandItem = new QWaylandQuickItem(q);

    connect(m_waylandItem, &QWaylandQuickItem::surfaceDestroyed, q, [this]() {
    
    
        // keep the buffer there to allow us to animate the window destruction
        m_waylandItem->setBufferLocked(true);
    });
}

• QWaylandQuickItem inherits QQuickItem and implements the updatePaintNode function. Classes that inherit QQuickItem can draw the content they want by implementing this virtual function. This code is relatively long. Basically, it takes out the Buffer from the Surface (that is, the View), makes it into a texture node and returns it. QT's underlying RenderThread will draw the node to the screen.

QSGNode *QWaylandQuickItem::updatePaintNode(QSGNode *oldNode, UpdatePaintNodeData *data)
{
    
    
    Q_D(QWaylandQuickItem);
    d->lastMatrix = data->transformNode->combinedMatrix();
    const bool bufferHasContent = d->view->currentBuffer().hasContent();

    if (d->view->isBufferLocked() && d->paintEnabled)
        return oldNode;

    if (!bufferHasContent || !d->paintEnabled || !surface()) {
    
    
        delete oldNode;
        return nullptr;
    }

    QWaylandBufferRef ref = d->view->currentBuffer();
    const bool invertY = ref.origin() == QWaylandSurface::OriginBottomLeft;
    const QRectF rect = invertY ? QRectF(0, height(), width(), -height())
                                : QRectF(0, 0, width(), height());

    if (ref.isSharedMemory()
#if QT_CONFIG(opengl)
            || bufferTypes[ref.bufferFormatEgl()].canProvideTexture
#endif
    ) {
    
    
#if QT_CONFIG(opengl)
        if (oldNode && !d->paintByProvider) {
    
    
            // Need to re-create a node
            delete oldNode;
            oldNode = nullptr;
        }
        d->paintByProvider = true;
#endif
        // This case could covered by the more general path below, but this is more efficient (especially when using ShaderEffect items).
        QSGSimpleTextureNode *node = static_cast<QSGSimpleTextureNode *>(oldNode);

        if (!node) {
    
    
            node = new QSGSimpleTextureNode();
            if (smooth())
                node->setFiltering(QSGTexture::Linear);
            d->newTexture = true;
        }

        if (!d->provider)
            d->provider = new QWaylandSurfaceTextureProvider();

        if (d->newTexture) {
    
    
            d->newTexture = false;
            d->provider->setBufferRef(this, ref);
            node->setTexture(d->provider->texture());
        }

        d->provider->setSmooth(smooth());
        node->setRect(rect);

        qreal scale = surface()->bufferScale();
        QRectF source = surface()->sourceGeometry();
        node->setSourceRect(QRectF(source.topLeft() * scale, source.size() * scale));

        return node;
    }

#if QT_CONFIG(opengl)
    Q_ASSERT(!d->provider);

    if (oldNode && d->paintByProvider) {
    
    
        // Need to re-create a node
        delete oldNode;
        oldNode = nullptr;
    }
    d->paintByProvider = false;

    QSGGeometryNode *node = static_cast<QSGGeometryNode *>(oldNode);

    if (!node) {
    
    
        node = new QSGGeometryNode;
        d->newTexture = true;
    }

    QSGGeometry *geometry = node->geometry();
    QWaylandBufferMaterial *material = static_cast<QWaylandBufferMaterial *>(node->material());

    if (!geometry)
        geometry = new QSGGeometry(QSGGeometry::defaultAttributes_TexturedPoint2D(), 4);

    if (!material)
        material = new QWaylandBufferMaterial(ref.bufferFormatEgl());

    if (d->newTexture) {
    
    
        d->newTexture = false;
        material->setBufferRef(this, ref);
    }

    const QSize surfaceSize = ref.size() / surface()->bufferScale();
    const QRectF sourceGeometry = surface()->sourceGeometry();
    const QRectF normalizedCoordinates =
            sourceGeometry.isValid()
            ? QRectF(sourceGeometry.x() / surfaceSize.width(),
                     sourceGeometry.y() / surfaceSize.height(),
                     sourceGeometry.width() / surfaceSize.width(),
                     sourceGeometry.height() / surfaceSize.height())
            : QRectF(0, 0, 1, 1);

    QSGGeometry::updateTexturedRectGeometry(geometry, rect, normalizedCoordinates);

    node->setGeometry(geometry);
    node->setFlag(QSGNode::OwnsGeometry, true);

    node->setMaterial(material);
    node->setFlag(QSGNode::OwnsMaterial, true);

    return node;
#else
    qCWarning(qLcWaylandCompositor) << "Without OpenGL support only shared memory textures are supported";
    return nullptr;
#endif // QT_CONFIG(opengl)
}