From 0 to 1 design general data large screen construction platform

Author: vivo Internet Big Data Team - Wang Lei

I. Introduction

For a long time, many product platforms have been trying to reduce the research and development threshold of GUI applications and improve production efficiency by means of visual construction. With the development of our business and the improvement of data construction, users' demands for data visualization are also increasing, and the big data screen is one of the display methods of data visualization. As a kind of big data display medium, it is widely used in Various exhibitions, company exhibition halls, press conferences, etc.

Compared with traditional manual customized charts and data dashboards, the emergence of a general-purpose large-screen construction platform can solve the problems of high application development and data maintenance costs caused by customized development and data dispersion. It can display various indicators in multi-directional, multi-angle, panoramic view, real-time monitoring, and dynamic at a glance.

This article will explain the overall design idea of ​​the general visualization building platform through the realization plan of the general large-screen construction capability of the agile BI platform.

2. Quickly understand the visual large screen

2.1 What is data visualization

From a technical point of view, the most intuitive is the front-end visualization framework: Echart, Antv, Chart.js, D3.js, Vega, etc. These libraries can help us quickly convert data into various forms of visual charts.

From a business perspective , its main significance lies in helping users analyze the trends and laws of different industries and scenarios more intuitively and holistically through the business process of data->chart combination->visualization page .

Therefore, in the data field, for complex and difficult to understand and huge data, the amount of information in the chart is much larger, which is also the most fundamental purpose of data visualization.

2.2 What are the parts of the large visual screen

It is mainly composed of visual components + event interaction + coordinate relationship. The effect is shown in the following figure:

2.3 The difference between the visual large screen and the common BI report kanban

Often students will ask, what is the difference between a large visual screen and a BI report kanban ?

Here is a brief introduction:

1. Large screens and report boards are only one of the ways of displaying BI. Large screens are mostly projected on display hardware of different sizes, while report boards are mostly displayed on the computer side.

2. The large screen pays more attention to the dynamic changes of data, which will have a strong visual experience and impact, and provide rich animation effects such as carousel animation and table scrolling. The report board pays more attention to interactive data exploration and analysis, such as scrolling up and down, sorting, filtering, chart switching, conditional warning, etc.

3. Design ideas

3.1 Technical selection

• Front-end framework: React family bucket (personal habit)

• Visualization framework: Echarts \ DataV-React (high encapsulation, easy expansion of json structure configuration items) D3.js (small granularity of visual elements, strong customization ability)

• Drag and drop plugin: dnd-kit (cross-component drag and drop for tree structure view)

• Layout plugin: React-Grid-Layout (grid free layout, modify source code, support drag and drop in multiple directions, free layout, lock zoom ratio, etc.)

3.2 Architecture Design

The following figure is the overall architecture design of our platform:

The entire large-screen construction platform includes four very important subsystems and modules:

• Visual Material Center: It is the most basic module of the entire platform. We define a standard DSL protocol on the open source chart library and self-developed visualization components. This protocol corresponds to the protocol for accessing the canvas editor. There are 40+ related components and the number of components is still growing.

• Canvas editor: It is the core and difficulty of building a platform. It supports functions such as page layout configuration, page interaction configuration, and component data configuration. It also supports the configuration of code snippets. It can also be called a low-code platform.

• Data center: It provides services dedicated to connecting different data sources, such as direct connection to MySQL, ClickHouse, Elasticsearch, Presto, etc., providing the raw data required for large-screen construction.

• Management Center: It is the background operation management module of the large screen, including the management functions of the large screen template management, the large screen release offline, access rights and other management functions.

3.3 Construction process

Through the above-mentioned large-screen components, we can analyze and summarize the main process of large-screen construction as shown in the following figure:

Fourth, the realization of core functions

Next, we will analyze the implementation of several core functions of the platform one by one:

1. Large screen adaptive layout

Background: Solve the problem of page confusion and realize large screen adaptation of various resolutions:

Thinking: The first thing we think of is the combination of the mainstream vh, vw, and rem on the mobile terminal, as well as two schemes such as font.js+rem. The first solution is to define the size of the parent through media queries, and then use rem as the unit for various CSS properties such as height, margin, and padding of the component, and inherit the unit (1vw) of the parent setting to achieve adaptive adaptation. The second solution is to refer to a third-party script, write code in main.js for calculation, and use rem for inheritance to achieve adaptation.

① vh, vw, rem combination

//vw vh单位 w3c的官方解释 vw：1% of viewport’s width vh：1% of viewport’s height
//例如，设计稿的宽度为1920px，则1vw=19.2px，为了方便计算，我们将html元素的font-size大小设置为100px，也就是5.208vw=100px。
body,html {
     font-size:5.208vw
}

② font.js + rem

//监听窗口的oversize事件，来动态计算根节点字体大小，再配合rem做适配
(function(doc, win) {
    const docEl = doc.documentElement
    const resizeEvt = 'orientationchange' in window ? 'orientationchange' : 'resize'
    const recalc = function() {
      let clientWidth = docEl.clientWidth
      if (!clientWidth) return
      docEl.style.fontSize = 100 * (clientWidth / 1920) + 'px'
    }
    if (!doc.addEventListener) return
    win.addEventListener(resizeEvt, recalc, false)
    doc.addEventListener('DOMContentLoaded', recalc, false)
})(document, window)

Defect: When the third-party plug-in we use in the big screen uses px as the unit, for example, the line-height is set to 20px, then it cannot adapt to the line height, and there will be confusion problems such as overlapping. Or we use the postcss-px2rem plug-in for global replacement, but in the process of use, we need to pay attention to the plug-ins that have been adapted, such as Ant Design, otherwise the introduced antd control will have the problem of style disorder

Solution: Use CSS3's zoom transform: scale(X, Y) property, mainly by monitoring the onresize event of the browser window. When the window size changes, we only need to calculate the actual width and height of the large-screen container. The corresponding zoom-in and zoom-out ratio can realize the adaptation of the layout. We also provide different layout adaptation effects, such as equal-height zooming, equal-width zooming, full-screen coverage, etc. Different zooming methods determine our calculation. Aspect ratio priority. Therefore, we are doing the reduction function of the canvas later, and this solution can also be used directly.

// 基于设置的设计稿尺寸 换算对应的宽高比
useEffect(() => {
    const wR = boxSize.width / viewWidth;
    const hR = boxSize.height / viewHeight;
 
    setBgScaleRatio(wR);
    setBgHeightScaleRatio(hR);
}, [boxSize, viewWidth, viewHeight]);
 
//根据等宽、等高、全屏等不同的缩放比例 计算scale值
const getScale = (proportion, x, y) => {
    if (proportion === 'radioWidth') {
        return `scaleX(${x})`
    }
    if (proportion === 'radioHeight') {
        return `scaleY(${y})`
    }
    return `scale(${x}, ${y})`
}

2. General development process design of large-screen components

Background: With the increase of visual components and the cumbersome and lengthy process of adding components, in order to avoid repeated wheel building and subsequent introduction of third-party components, it is necessary to formulate a general component development process:

Design idea: component = component main body + schema component configuration protocol layer + component definition layer (type, affiliation, initialization width, etc.)

① component The main body of the component:

• Visualization framework selection: The mainstream visualization libraries in the industry include Echart, Antv, Chart.js, D3.js, Vega, and DataV-React. Based on the universality and customization requirements of visualization, we chose Echart and DataV-React as the basic components. For the development framework, we chose D3.js with smaller visualization granularity in the face of custom components with higher customization requirements.

• Encapsulate common Echarts components (initialization, event registration, instance deregistration, etc.):

// initialization echarts
const renderNewEcharts = () => {
    // 1. new echarts instance
    const echartObj = updateEChartsOption();
    // 2. bind events
    bindEvents(echartObj, onEvents || {});
    // 3. on chart ready
    if (typeof onChartReady === 'function') onChartReady(echartObj);
    // 4. on resize
    echartObj.resize();
};
 
// bind the events
const bindEvents = (instance, events) => {
    const _bindEvent = (eventName, func) => {
       instance.on(eventName, (param) => {
           func(param, instance);
       });
    };
 
    // loop and bind
    for (const eventName in events) {
        if (Object.prototype.hasOwnProperty.call(events, eventName)) {
            _bindEvent(eventName, events[eventName]);
        }
    }
};
 
// dispose echarts and clear size-sensor
const dispose = () => {
    if ($chartEl.current) {
       clear($chartEl.current);
       // dispose echarts instance
       (echartsLib || echarts).dispose($chartEl.current);
    }
};

• Encapsulate common DataV components (DataV-React, custom and other component entrances, unified responsible for configuration, data collection, monitoring resize)

const DataV: React.FC<DataVProps> = (props) => {
    const { config } = props;
    const [renderCounter, setRenderCounter] = useState(0);
    const $dataVWarpEl = useRef(null);
    const $componentEl = useRef(null);
 
    useEffect(() => {
        // 绑定容器size监听
        const resizefunc = debounce(() => {
            $componentEl.resize();
        }, 500)
       // fixme
       addResizeListener($dataVWarpEl.current, resizefunc);
       return () => {
           // 清除订阅
           removeResizeListener($dataVWarpEl.current, resizefunc);
       };
    }, []);
 
    return (
        <DataVWarp ref={$dataVWarpEl}>
            <CompRender config={config} ref={$componentEl} />
        </DataVWarp>
    );
};

② schema component configuration protocol layer + component definition layer (type, affiliation, initialization width, etc.)

We customized a set of schema component DSL, the structure protocol layer. Through the DSL, the configuration protocol of the component is agreed, including the editable attributes, editing type, initial value, etc. of the component. The reason for defining a consistent protocol layer is mainly to facilitate the later component expansion and the configuration to be moved later.

• JSON Schema design:

{
    headerGroupName: '公共配置',                         //配置所属类型
    headerGroupKey: 'widget',                           //配置所属类型key值 相同的key值都归属一类
    name: '标题名称',                                    //属性名称
    valueType: ['string'],                              //属性值类型
    optionLabels: [],                                   //服务下拉列表、多选框等控件的标签名
    optionValues: [],                                   //服务下拉列表、多选框等控件的标签值
    tip: false,                                         //配置项的 Tooltip 注解
    ui: ['input'],                                      //使用的控件类型
    class: false,                                       //控件类名，定制控件样式
    css: { width: '50%'},                               //修改控件样式
    dependencies: ['widget,title.show,true'],           //属性之间的联动，规则['配置所属类型, 属性key, 属性值']
    depContext: DepCommonShowState,                     //属性之间的校验回调方法
    compShow: ['line'],                                 //哪些组件可配置
    dataV: { key: 'title.text', value: '' },            //配置的key值和默认value值
},

• Form DSL design:

Benefits: The above is our customized DSL structure protocol layer. Users only need to fill in the Excel form to create dynamic forms, and realize the functions of component configuration item classification, configuration reuse, linkage between configuration items, and attribute annotation. At present, the property configurator already supports 15 commonly used configuration UI controls. Through the customized DSL structure protocol layer, the initialization of the configuration interface of the component can be quickly completed to prepare for the subsequent planning of the component material center.

3. Drag and drop implementation

Background: The React-Grid-Layout drag-and-drop plugin does not support free layout and drag-and-drop of components at different latitudes:

Solution: By analyzing the source code, the drag events of different latitudes and the drag target collision events were rewritten, and the functions of locking the aspect ratio and rotating transparency were also expanded.

Source code analysis:

The resize scaling feature is enhanced (optimized), while dragging, in addition to modifying the width and height of the container, the coordinate position of the component is also dynamically adjusted

// CSS Transforms support (default)
if (useCSSTransforms) {
    if (activeResize) {
        const { width, height, handle } = activeResize;
        const clonePos = { ...pos };
        if (["w", "nw", "sw"].includes(handle)) {
            clonePos.left -= clonePos.width - width;
        }
        if (["n", "nw", "ne"].includes(handle)) {
            clonePos.top -= clonePos.height - height;
        }
        style = setTransform(clonePos, this.props.angle);
    } else {
        style = setTransform(pos, this.props.angle);
    }
}

Stacked display, free layout (optimization), by controlling whether the layout is compressed, dynamically adjusting the level zIndex of the drag target to achieve multi-layer component operation interaction and free positioning.

// 每次拖拽时zIndex要在当前最大zIndex基础上 + 1，并返回给组件使用
const getAfterMaxZIndex = useCallback(i => {
    if (i === curDragItemI) {
        return;
    }
    setCurDragItemI(i);
    setMaxZIndex(maxZIndex => maxZIndex + 1);
    return maxZIndex;
}, []);

Effect display after transformation

4. Big screen status push

Background: The post-maintenance of the large screen requires the self-update of version release and the offline of the large screen. At this time, a set of message notification mechanism is required to control the running status of the large screen through commands.

Solution: Based on the websocket communication mechanism, a long link is established, the heartbeat and reconnection mechanism is realized, and the large screen after the online release is updated or offline management in real time.

5. Effect preview

6. Summary

This article analyzes and explains how to design and develop a general data screen construction platform through technical ideas such as visual page construction, no/low code platform, and Schema dynamic form.

The current design scheme basically meets the requirements for building the core capabilities of the big data screen. If we want to build a platform with a larger screen that is more expressive and can meet more scenarios, we need to further improve the scalability of the platform and improve the overall material ecology. The specific plans are as follows:

• Enrich and expand large-screen components & configuration capabilities to cover visualization scenarios in different industries.

• The construction of the visual material platform has precipitated excellent visual components and large-screen template materials.

• 3D and dynamic rendering engine development and implementation.

{{o.name}}

{{m.name}}