About the Author
ck, Ctrip back-end development expert, focusing on technical architecture, high concurrency, performance tuning and other fields;
Geralt, Ctrip front-end development expert, focuses on front-end framework and performance optimization;
Kaoru, senior front-end development engineer at Ctrip, focuses on front-end performance and development tools;
Overview
PGClowcode platform is a low-code platform built by Ctrip Market Content PGC team mainly for back-end page development. The first version was launched in March 23. As of October, the platform has 100+ users and 130+ applications have been developed on the platform. and 180+ pages. This article will mainly introduce the background of the team's adoption of the low-code platform, solution research, problems encountered during the implementation process, and solutions. It will also give a general introduction to the capabilities provided by the low-code platform.
1. Research background
1.1 Why you need a low-code platform
Software products usually consist of clients (Apps, mini-programs, web pages) and operation backends. Their entire life cycle requires constant updates and iterations. In actual iterative development, the following problems often occur:
Backend demand has low priority, and scheduling is often delayed. Simple data operation platforms such as configuration centers are often used instead, which is not only unfriendly to operators, but also creates great production risks.
The workload of front-end and back-end development involved in iterative requirements is unbalanced. There are often more back-end demands, but the front-end resources are insufficient. Even if the server resources are free, they cannot help support the back-end demands.
The backend technologies and components of different businesses are largely the same, and each business repeatedly reinvents the wheel, wasting time and resources.
When developers develop tool sites, they need to pay a greater price for the page part of the work.
We have also tried many solutions to these problems, such as requiring developers to be full-stack, stripping off common components, etc., but the results are not ideal. After full understanding and analysis, building a low-code or zero-code platform can help to a great extent. to solve these problems.
1.2 What kind of low-code platform is needed?
There are many types of low-code platforms, which one do we need? After breaking down the above issues and fully investigating them, our low-code platform should meet the following requirements:
Page construction:
The interface is friendly and visual, allowing R&D and related personnel to quickly build UI through drag and drop.
Page logic only requires writing a small amount of simple code or no code
Supports component libraries that meet daily needs
Deployment and operation:
It has complete continuous delivery functions such as development, deployment, operation and maintenance, etc. It is best to complete it with one click or automatically.
Safety compliance:
Support permission management mechanism to ensure system security
It has a complete release approval mechanism to strictly ensure development quality and production safety.
Compatibility:
Can embed existing backend pages, and embed the pages developed by existing backends with as little modification as possible
In terms of scalability:
Support user self-service component development and promote it on the platform
2. Current situation analysis
2.1 Current status of the industry
There are currently more than a hundred low-code or zero-code products at home and abroad, including commercial platforms and open source projects. They are used within enterprises to develop internal systems such as various operational backends, data panels, and office systems, and provide product management on the B-side. , advertising placement, store construction and other enterprise services are widely used on the C side to build event pages, promotion channels, advertising channels and other types of products. It not only saves a lot of development costs for enterprises and generates huge commercial value, but also provides users with Provides a variety of software products.
Although there are many types of low-code projects, each type of low-code project often has specific business attributes. In different industries, different companies may need to customize different components and processes, so there are few on the market that can be suitable for all scenarios. Platform, and few companies are willing to build a universal platform. The following is an analysis of the currently popular products:
2.2 Product Analysis
Alibaba low-code engine
LowCodeEngine (Alibaba low-code engine) is one of the most well-known low-code products in China. It fully implements the "Low-code Engine Construction Protocol Specification" and "Low-Code Engine Material Protocol Specification". It has agreed upon through a complete protocol stack It develops, uses, expands, and circulates various materials, and refines the characteristics of an enterprise-level low-code platform. It follows the concept of expansion-oriented design and pursues the design principles of the smallest kernel and the strongest ecology. The kernel engine is designed.
At present, its ecosystem is quite complete, and it provides very detailed documents and use cases. There are also a large number of open source demos, which are relatively convenient to use. However, it does not provide a complete platform code. Users need to build a user-oriented platform and service system based on its core, which has considerable development costs.
Tencent low-code platform
Tencent's low-code products include the tmagic-editor open source project that builds mobile H5 pages and the Wuji low-code commercial platform that builds management pages. Among them, tmagic-editor provides complete platform code, and users can use the entire project in the open source community. Pull to local deployment and use, it has the expansion capabilities of custom components, plug-ins, editors, etc. It has rich built-in components, provides a friendly visual interface, and develops pages through drag and drop + configuration.
However, it is limited to the construction of mobile H5 pages and cannot meet the needs of PC page development, which greatly limits the scope of use. The Wuji low-code platform is a commercial paid-use product for enterprises. It has strong development capabilities on the management side and even combines AI capabilities to provide a wealth of solutions for enterprises. However, it is not currently open source.
Open source low-code platform
Search lowcode keywords on github and you can see a lot of projects. The technologies they use, the shape of the project, the number of stars, the level of activity, and the usage scenarios are all different. Some provide a complete visual interface, and some require Generate pages or projects through configuration files or calling interfaces. In short, it takes a lot of time to compare and analyze to find projects that meet your needs.
Summarize
To sum up, we finally decided to choose a relatively high-level open source project (appsmith) as a blueprint, and then transformed and developed the PGClowcode platform. ˆ
3. Introduction to platform functions
The functions of the PGClowcode platform mainly include page construction and component development platform.
Page construction includes common functions such as development, preview, testing, publishing, rollback, backup, and recovery. On the basis of these functions, functions such as "visual drag and drop", "multi-user collaborative development", "import and export", "multiple data sources", and "notifications" have been added to form a A fairly complete development system has been established. The built products can be used in combination by embedding applications or pages on the platform into the existing backend, or conversely, embedding existing backend pages into the platform.
The component development platform is an expansion of page building capabilities, supporting the construction of local projects through the CLI and custom development of new components to meet more complex business needs. (This feature is under development and will be available soon)
Due to limited space, several main functions will be introduced below.
3.1 User and permission management
The platform has its own user management system. In order to connect with Ctrip's account system, it has integrated OIDC domain account authentication. Users do not need to register an additional account and only need to log in with a Ctrip domain account.
User permissions have been fully split. All functions of the platform are open to every user, but permissions are controlled for user private data. The smallest unit of permission definition is a workspace, and users can have multiple workspaces. Each workspace defines five permission groups: administrator (admin), developer (developer), user (viewer), tester (tester), and reviewer (reviewer). Each permission group has different management permissions for the resources under the workspace. These resources include data sources, applications, pages, etc. The admin can assign different permission groups to users in the workspace, thereby improving application development, publishing and other processes. to manage. Please refer to the following table for the specific permission distribution of each permission group:
3.2 Visual application development
The difference between the process of traditional back-end development and the use of low-code platform for back-end development is shown in the figure below:
In the traditional backend development process, developers need to build their own development environment and introduce front-end component libraries such as Ant Design. For the same functions, they need to extract components themselves, which results in inefficient development.
The PGClowcode low-code platform provides a visual drag-and-drop panel that supports complex page layouts. The component bar supports 40+ common components and can be used in combination.
In terms of page drawing, by dragging it into the artboard, adjusting the position and layout, and completing the interface design in a few simple steps, what you see is what you get. The same functions can be copied and pasted in the canvas, and the application itself also supports import and export functions to facilitate project copying. Development becomes flexible and efficient, avoiding bugs caused by some basic construction, achieving the effect of cost reduction and efficiency improvement.
In terms of component attribute value setting, complex logic binding can be performed through visual input or custom JS code, and it also supports writing JS code to complete complex interaction logic. The platform has a variety of built-in js libraries that can bind data to components. You can immediately see the effect of data rendering in the development state, allowing you to self-test while developing in the preview state.
3.3 Process management
It is very convenient to transfer applications from development state to test state to release state. The platform has designed different roles to simplify the testing and review process. After logging in, each role can see the different status of the application. After the application is developed and approved, it will automatically flow to the next status. It only requires a few simple processes to complete.
1) Developers (development status): build and develop pages according to needs, and then publish them to the test environment;
2) Tester (testing state): After testing the page to ensure that it can meet the needs and there are no quality problems, click Publish to submit the publishing application;
3) Reviewer (released): Find the review application in the "Waiting for review" list. Once approved, the application is automatically released.
3.4 Backup and restore
The development platform supports the backup of application data and the restoration of historical version data. The platform adopts an automatic saving design during development. Since multiple people are prone to overwriting or operating conflicts when developing at the same time, in order to reduce the rework costs caused by this problem, we have designed backup and restore functions.
Just like normal applications, users can back up the development state of each stable version to the system. If they need to return to a stable version in subsequent operations, they can directly select the target version to restore.
The figure below shows the backup and restore operation interface.
4. Architecture and Technology
The PGClowcode platform adopts a front-end and back-end separation architecture. The front-end uses the react technology stack and integrates Ctrip's various public frameworks and components. It relies on Ctrip's CI/CD platform to achieve continuous development and delivery capabilities. The server uses the spring webflux framework, integrating technical frameworks such as cat, clog (Ctrip log system), mongo, credis (Ctrip redis client), qconfig (Ctrip Configuration Center middleware, hereinafter referred to as QC), qmq (Ctrip MQ middleware), etc. , fully integrated into Ctrip's service technology stack, can be automatically compiled and packaged through gitlab, and released on captain (Ctrip publishing platform).
4.1 Architecture
As shown in the figure above, the overall architecture of the PGClowcode platform is divided into application layer, infrastructure, service layer and data layer.
The application layer is the two platform systems of the terminal, which mainly includes a user-oriented low-code development platform and a developer-oriented component development platform.
The infrastructure mainly includes the front-end basic framework, data flow control, and abstracted front-end visual components, pages, editors and other concepts. The basic framework mainly relies on React App and canvas technology to interact with data through the axios library and the server. It uses Redux and related plug-ins to control the data flow of the entire platform, and finally displays it into user-visible components, pages, editors and other UI modules.
The service layer is mainly composed of the web layer, service layer and data access layer, and mainly provides services such as permission verification, process control, plug-in management, message notification, and data access. The service adopts the design of microservice architecture and is divided into domain services, notification services and plug-in services according to different fields and functions.
Domain services are subdivided into different modules according to different models, each completing an independent and single function; notification services are mainly used for email, trippal, SMS and other message notifications; plug-in services mainly provide plug-in loading, initialization, calling, uninstallation and other related related Functional services. Data access is at the lower level of the core service and implements the functions of accessing multiple data sources, data processing, and verification.
The data layer is mainly used to store metadata, application data, plug-in data, etc. The backup data of the application is stored in QC, and cross-environment release is achieved through QC.
The following two sections mainly elaborate on the two core technologies of the platform, "visual drag-and-drop implementation of components" and "application development-deployment implementation".
4.2 Visual drag-and-drop implementation of components
Visual drag-and-drop components are a basic function of the low-code platform. In this platform, users can select any component from the component library on the left and drag it to the middle editing area and change its size.
When implementing the above drag and drop function, we will face several problems:
1) During the process of dragging components, the position of the component will change in real time. A large number of nested DOM elements will change their positions at the same time, which will frequently trigger redrawing of the browser, resulting in performance consumption.
2) Manual drag-and-drop arrangement makes it difficult to ensure the alignment and layout of components, and the final effect of the page is difficult to achieve the regularity of the code-implemented page.
For the above problem 1, the platform's solution is to rely on canvas technology. When the component changes its position or size, it hides the component actually rendered on the page, renders a layer of canvas on the top layer of the editing area, and draws an equivalent image at the original component position. Color blocks of different sizes are used to replace the original components and to indicate to the user. The characteristics of the canvas are used to handle changes in the position and size of the components. After the user completes the dragging action, the color block is redrawn based on its final position and size in the canvas. Component once and hide the canvas to show the final effect of the component.
As can be seen from the above description, using canvas can greatly reduce the number of browser redraws and reduce performance consumption; selecting color blocks as drawing objects instead of original components not only reduces the difficulty of technical implementation, This further reduces the performance consumption of canvas drawing graphics.
For problem 2, the platform’s solution is the lattice system. When the user drags the component to the editing area to trigger rendering of the canvas, a layer of lattice points will be drawn on the page and evenly tiled on the canvas. When the user drags the component on the canvas, When you drag the color block to change its position or size, a dotted border of the same size will be drawn around the color block. The border will be dynamically positioned at the appropriate adjacent matrix point based on the current position and size of the color block. In the lattice point system, the distance between two adjacent lattice points in the horizontal and vertical directions is the smallest unit of the length and width of the module. The four corners of the module can only be positioned on the lattice points. It can be seen that during the dragging process, the position and size of the components are within certain limits, which can ensure the regularity of the final drawn page.
The following is a schematic diagram of a complete component dragging process:
1) There is no drag and drop operation on the page. The main editing area only displays the static state of the component, which is in the display state at this time.
2) When the user starts dragging the component in order to change its position and size, the dynamic state is the operation state. The operation state can be further refined into three states: dragging start, dragging, and dragging end.
When the user starts dragging the component, the page will change from the display state to the operation state. When dragging starts, a canvas is drawn in the editing area and the dots are laid. The original component becomes invisible. A color block of the same size and a dotted border are drawn at the same position of the original component in the canvas; during the dragging process, the color block will follow the dragging process. When the mouse moves to change the position or size, its outer dotted border will also move or change size with the color block, and will be positioned in real time at the most adjacent matrix point of the current position of the color block; when dragging ends, the final size and position of the current border will be , redraw the original component and hide the canvas, dot system, color block and dotted border; the page changes from the operation state to the display state, showing the latest status of the component.
4.3 Application development-deployment implementation
The technical implementation of application development and deployment is mainly divided into three parts: application data structure, data flow and collaboration of multiple roles. Finally, a brief introduction is given to application data backup and restoration.
4.3.1 Data structure
The final product of most current mainstream low-code platforms may be code, but the final product of the PGClowcode platform is data, including application information, page information, components, relationships between components, actions, data sources, etc., all in the form of data Stored on db, due to the very complex relationships between various entities such as page layout, component nesting, and function binding, traditional relational databases cannot guarantee the stability of the data structure, so mongodb is used as the database. Application-related collections mainly include plugin, workspace, datasource, application, page, action, and actioncollection. They form the entire application system through the relationship in the figure below.
Plugin is mainly used to define the metadata information of plug-ins supported by the platform, including attributes such as type, plug-in package path, parameters, and status.
Workspace is the workspace for application development. It defines the user composition, user permissions, data sources and supported plug-in collections in the space.
application defines basic information such as the application's name, theme, icon, status, etc. In addition, for the convenience of query, some page information is also redundant.
Page defines the page name, layout, components, relationship between components, binding relationship between components and functions, etc.
Datasource defines basic information for external data access, such as restapi, es, mongodb and other necessary attributes for external data access. In order to avoid repeated configuration, its scope is at the workspace level.
Action defines the specific configuration data for external data access. It must rely on datasource, such as restapi interface call. Datasource configures the domain name of the service and whether real-time authentication is required, while action configures the path, parameters, and operation mode, which is different from datasource. is that its scope is the page.
actionCollection is a collection of actions or js functions.
4.3.2 Data flow
Application data mainly flows between different storage media and different environments. The platform currently designs three sets of environments: FAT, UAT, and PRO. The platform realizes data flow through QC's cross-environment mechanism.
As shown in the figure above, developers develop applications on FAT, and the application data is stored in DB in draft state. After development is completed, the draft state data is exported to QC's FAT environment. The service monitors QC's data changes and copies the draft state to the release state. , for testing, you can see the application submitted for development on the test page. After the test is completed, submit the UAT release application. The service will export the released data to the UAT environment of QC. At this time, the reviewer will receive a notification to be reviewed.
After entering the platform and reviewing the application to be released, the service of the UAT environment monitors the QC data changes, imports the data into the DB, and sets the application data status to the released state. Then you can see the released application on the UAT test page. After the UAT test is completed, apply for release to production (PRO). The UAT service will export the released application data to the QC PRO environment. When the reviewer passes the application, the QC PRO application file will be released. The PRO service will monitor the data changes and apply the data. Import it into DB and set it to the release state. At this time, the application development-deployment process ends, and users can use it normally on the user platform in the production environment.
4.3.3 Multi-role collaboration
Multiple roles are designed for the application development, testing, and delivery platform. Each role can perform its own responsibilities during the entire requirements development process to ensure that the application can be delivered continuously, stably, and efficiently iteratively, as shown in the figure below.
The platform distinguishes each role by defining multiple permission groups. These permission groups will be created when the workspace is created. Each permission group defines its own permissions. There is a policies field in each resource managed by permissions. , which stores a collection of types and permission group IDs that can be operated, and will go through permission verification when users query and operate.
With the definition of roles, collaboration among application developers becomes much simpler. For example, testers can be automatically notified to intervene when an application is released for testing, and reviewers can be automatically notified to participate in the review when a production application is submitted.
4.3.4 Backup and restore
In order to facilitate the simultaneous development of multiple people during the development process, the platform has designed a backup and restore function. When the user clicks Backup, the service will export the current draft state data to QC. Clicking Restore will import the QC data into the service, overwriting the current draft state in DB. Data, thanks to QC's version management function, each backup data will be stored, so users can restore to any version in history.
As shown in the figure above, a backup v1 is added at time T1, and the backup data of version v1 exists in QC at time T2. If a backup v2 is added at time T2, there will be two versions of v1 and v2 in QC at time T3. If the version in the DB is restored to v1, the data in the DB will be restored to v1. At the same time, a v3 version will be uploaded to QC. In fact, the v3 and v1 data are the same, which is equivalent to switching the baseline of the current data to the latest version, all subsequent operations will be done on the basis of this version. If you need to restore to this baseline again, you can quickly complete it.
5. Planning and Prospects
At present, the PGC low-code platform has very complete functions, has basically completed the expected goals, and also generated greater value. However, our expectations for it are by no means limited to this, and we have established a stable support team and formulated a With a clear plan, existing functions and processes will be continuously improved in subsequent iterative development, and more powerful and convenient functions will be added based on actual needs and industry platform research.
5.1 Build a component expansion platform
The original components of the platform are relatively basic component libraries. In the future, it will be difficult to meet increasingly complex business needs, so the need for custom components will gradually become more prominent. This platform supports custom components based on the Appsmith framework, but the original Custom functions have the following problems:
1) The original custom component function needs to rely on a complete platform project and develop new components under its widget folder. The project code is large and slow to start, and it is not feasible to submit platform projects with frequent changes for the purpose of developing components. Conducive to the release and management of platform projects.
2) The original custom component function is not suitable for multi-department custom component scenarios. There is no relevant permission management system. All custom components will be displayed on the page, which will cause the component library to become bloated and increase over time. Find out how effortful it is to use a component when editing a page.
In order to solve the above problems, we will extract the relevant code from the main project and build a set of independent component development projects to achieve the purpose of separation from the main platform project, pure code, and quick start of the project. At the same time, a permission management system for custom components will be built to better manage custom components developed by various departments.
5.2 Establish a template library
Ctrl+C and Ctrl+V may be the most frequently used key combinations by developers, causing some keyboards to be either "unrecognizable" or "hemiplegic". Just imagine, if we take out the function of generating a deployment page with one click, can it make "long-term use" possible? Is a keyboard that has withstood the test of time still "lasts forever"? Yes, this is our next goal.
The template of the low-code platform refers to summarizing some common solutions based on long-term accumulated experience, and refining them into pages or applications that all users can directly use, and including them in the template library. When other users on the platform need to use similar When applying or page, you only need to find a suitable template, and the page or application can be generated with one click. It can even be used immediately without any modification. Later, the team can be allowed to establish its own template library, and each member can build his own template specifically for the team's use.
[Recommended reading]
Improve efficiency and user experience, Ctrip ticket activity product structure evolution
Ctrip International Air Ticket Architecture Reconstruction Practice
Low cost and small error, Ctrip’s practice of Kafka-based Serverless delay queue
“Ctrip Technology” public account
Share, communicate, grow